[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:49:34 dist-test-slave-p5bw krb5kdc[6759](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:49:34 dist-test-slave-p5bw krb5kdc[6759](info): set up 2 sockets
Sep 28 23:49:34 dist-test-slave-p5bw krb5kdc[6759](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 23:49:35.601392 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:49:35 dist-test-slave-p5bw krb5kdc[6759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:49:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:49:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:49:35.725294 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:38601
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:38601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:35.832358 6772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:35.832610 6772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:35.832679 6772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:35.836632 6772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:49:35.836684 6772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:35.836704 6772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:35.836758 6772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:49:35.836792 6772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:49:35.842047 6772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:38601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:38601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:35.843101 6772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:35.844154 6772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:35.852919 6778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:35.852932 6780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:35.852983 6772 server_base.cc:1034] running on GCE node
W20260928 23:49:35.852932 6777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:35.853792 6772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:35.854903 6772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:35.856096 6772 hybrid_clock.cc:648] HybridClock initialized: now 1790639375856063 us; error 45 us; skew 500 ppm
Sep 28 23:49:35 dist-test-slave-p5bw krb5kdc[6759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639375, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:35.858824 6772 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:49:35.859791 6772 webserver.cc:460] Webserver started at http://127.6.152.62:37927/ using document root <none> and password file <none>
I20260928 23:49:35.860327 6772 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:35.860384 6772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:35.860543 6772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:35.862604 6772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "e74130c28588480c99c42a702349dca6"
format_stamp: "Formatted at 2026-09-28 23:49:35 on dist-test-slave-p5bw"
server_key: "9802e0dc46f6de84cf26cc1ef34da690"
server_key_iv: "4429266b3dd3dd470909663746a2f0a8"
server_key_version: "encryptionkey@0"
I20260928 23:49:35.863164 6772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "e74130c28588480c99c42a702349dca6"
format_stamp: "Formatted at 2026-09-28 23:49:35 on dist-test-slave-p5bw"
server_key: "9802e0dc46f6de84cf26cc1ef34da690"
server_key_iv: "4429266b3dd3dd470909663746a2f0a8"
server_key_version: "encryptionkey@0"
I20260928 23:49:35.866945 6772 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:49:35.869580 6787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:35.870944 6772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:49:35.871095 6772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "e74130c28588480c99c42a702349dca6"
format_stamp: "Formatted at 2026-09-28 23:49:35 on dist-test-slave-p5bw"
server_key: "9802e0dc46f6de84cf26cc1ef34da690"
server_key_iv: "4429266b3dd3dd470909663746a2f0a8"
server_key_version: "encryptionkey@0"
I20260928 23:49:35.871201 6772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:35.892571 6772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:35.895919 6772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:35.896157 6772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:35.903429 6772 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:38601
I20260928 23:49:35.903446 6839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:38601 every 8 connection(s)
I20260928 23:49:35.904407 6772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:49:35.907066 6840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:35.911360 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 6772
I20260928 23:49:35.911525 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:49:35.912460 6840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e74130c28588480c99c42a702349dca6: Bootstrap starting.
I20260928 23:49:35.912937 6752 external_mini_cluster.cc:1428] Setting key b228caf66cdcf4aee50ce634d9678cba
I20260928 23:49:35.914958 6840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e74130c28588480c99c42a702349dca6: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:35.915890 6840 log.cc:824] T 00000000000000000000000000000000 P e74130c28588480c99c42a702349dca6: Log is configured to *not* fsync() on all Append() calls
I20260928 23:49:35.918006 6840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e74130c28588480c99c42a702349dca6: No bootstrap required, opened a new log
I20260928 23:49:35.921700 6840 raft_consensus.cc:348] T 00000000000000000000000000000000 P e74130c28588480c99c42a702349dca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e74130c28588480c99c42a702349dca6" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38601 } }
I20260928 23:49:35.921948 6840 raft_consensus.cc:374] T 00000000000000000000000000000000 P e74130c28588480c99c42a702349dca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:35.922032 6840 raft_consensus.cc:729] T 00000000000000000000000000000000 P e74130c28588480c99c42a702349dca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e74130c28588480c99c42a702349dca6, State: Initialized, Role: FOLLOWER
I20260928 23:49:35.922587 6840 consensus_queue.cc:260] T 00000000000000000000000000000000 P e74130c28588480c99c42a702349dca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e74130c28588480c99c42a702349dca6" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38601 } }
I20260928 23:49:35.922742 6840 raft_consensus.cc:388] T 00000000000000000000000000000000 P e74130c28588480c99c42a702349dca6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:49:35.922842 6840 raft_consensus.cc:482] T 00000000000000000000000000000000 P e74130c28588480c99c42a702349dca6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:49:35.922969 6840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e74130c28588480c99c42a702349dca6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:35.923864 6840 raft_consensus.cc:504] T 00000000000000000000000000000000 P e74130c28588480c99c42a702349dca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e74130c28588480c99c42a702349dca6" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38601 } }
I20260928 23:49:35.924198 6840 leader_election.cc:302] T 00000000000000000000000000000000 P e74130c28588480c99c42a702349dca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e74130c28588480c99c42a702349dca6; no voters:
I20260928 23:49:35.924541 6840 leader_election.cc:288] T 00000000000000000000000000000000 P e74130c28588480c99c42a702349dca6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:49:35.924649 6845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e74130c28588480c99c42a702349dca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:49:35.924932 6845 raft_consensus.cc:686] T 00000000000000000000000000000000 P e74130c28588480c99c42a702349dca6 [term 1 LEADER]: Becoming Leader. State: Replica: e74130c28588480c99c42a702349dca6, State: Running, Role: LEADER
I20260928 23:49:35.925285 6845 consensus_queue.cc:237] T 00000000000000000000000000000000 P e74130c28588480c99c42a702349dca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e74130c28588480c99c42a702349dca6" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38601 } }
I20260928 23:49:35.925788 6840 sys_catalog.cc:564] T 00000000000000000000000000000000 P e74130c28588480c99c42a702349dca6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:49:35.926410 6845 sys_catalog.cc:455] T 00000000000000000000000000000000 P e74130c28588480c99c42a702349dca6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e74130c28588480c99c42a702349dca6. Latest consensus state: current_term: 1 leader_uuid: "e74130c28588480c99c42a702349dca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e74130c28588480c99c42a702349dca6" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38601 } } }
I20260928 23:49:35.926611 6845 sys_catalog.cc:458] T 00000000000000000000000000000000 P e74130c28588480c99c42a702349dca6 [sys.catalog]: This master's current role is: LEADER
I20260928 23:49:35.926820 6846 sys_catalog.cc:455] T 00000000000000000000000000000000 P e74130c28588480c99c42a702349dca6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e74130c28588480c99c42a702349dca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e74130c28588480c99c42a702349dca6" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38601 } } }
I20260928 23:49:35.926975 6846 sys_catalog.cc:458] T 00000000000000000000000000000000 P e74130c28588480c99c42a702349dca6 [sys.catalog]: This master's current role is: LEADER
I20260928 23:49:35.927086 6853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:49:35.929994 6853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 23:49:35 dist-test-slave-p5bw krb5kdc[6759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:35.935899 6853 catalog_manager.cc:1347] Generated new cluster ID: d380418e1b6b4e4ba110b6d59f37cbd9
I20260928 23:49:35.936009 6853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:49:35.939445 6842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:35.915742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37658 (local address 127.6.152.62:38601)
0928 23:49:35.916157 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:35.916172 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:35.917508 (+ 1336us) server_negotiation.cc:407] Connection header received
0928 23:49:35.921344 (+ 3836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:35.921360 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:35.921660 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:35.921984 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:35.924456 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:35.925941 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:35.927804 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:35.928650 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:35.928845 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:35.933848 (+ 5003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:35.933854 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:35.933866 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:35.933892 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:35.936115 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:35.936626 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:35.936632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:35.936637 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:35.936785 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:35.937183 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:35.937187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:35.937189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:35.937582 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:35.937810 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:35.938333 (+ 523us) server_negotiation.cc:299] Negotiation successful
0928 23:49:35.938604 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":137,"threads_started":1}
I20260928 23:49:35.948210 6853 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:49:35.949167 6853 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:49:35.959524 6853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e74130c28588480c99c42a702349dca6: Generated new TSK 0
I20260928 23:49:35.960218 6853 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
I20260928 23:49:35.992663 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:38601
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:36.140394 6866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:36.140643 6866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:36.140687 6866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:36.144672 6866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:36.144726 6866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:36.144827 6866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:49:36.149981 6866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:38601
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:36.150992 6866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:36.151978 6866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:36.161417 6874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:36.161427 6871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:36.161474 6866 server_base.cc:1034] running on GCE node
W20260928 23:49:36.161451 6872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:36.161996 6866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:36.162611 6866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:36.163792 6866 hybrid_clock.cc:648] HybridClock initialized: now 1790639376163769 us; error 35 us; skew 500 ppm
Sep 28 23:49:36 dist-test-slave-p5bw krb5kdc[6759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639376, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:36.166456 6866 init.cc:377] Logged in from keytab as kudu/127.6.152.1@KRBTEST.COM (short username kudu)
I20260928 23:49:36.167225 6866 webserver.cc:460] Webserver started at http://127.6.152.1:42131/ using document root <none> and password file <none>
I20260928 23:49:36.167721 6866 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:36.167776 6866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:36.167927 6866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:36.169924 6866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "72d86dae22154b119264b5dac2cf21cf"
format_stamp: "Formatted at 2026-09-28 23:49:36 on dist-test-slave-p5bw"
server_key: "9ddf910ec9b075ea8b80ed6734506f6a"
server_key_iv: "9fdc85747e97f7a1217ad660a2f12a55"
server_key_version: "encryptionkey@0"
I20260928 23:49:36.170421 6866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "72d86dae22154b119264b5dac2cf21cf"
format_stamp: "Formatted at 2026-09-28 23:49:36 on dist-test-slave-p5bw"
server_key: "9ddf910ec9b075ea8b80ed6734506f6a"
server_key_iv: "9fdc85747e97f7a1217ad660a2f12a55"
server_key_version: "encryptionkey@0"
I20260928 23:49:36.173867 6866 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260928 23:49:36.176249 6881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:36.177491 6866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 23:49:36.177593 6866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "72d86dae22154b119264b5dac2cf21cf"
format_stamp: "Formatted at 2026-09-28 23:49:36 on dist-test-slave-p5bw"
server_key: "9ddf910ec9b075ea8b80ed6734506f6a"
server_key_iv: "9fdc85747e97f7a1217ad660a2f12a55"
server_key_version: "encryptionkey@0"
I20260928 23:49:36.177722 6866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:36.189529 6866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:36.192194 6866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:36.192381 6866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:36.193001 6866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:49:36.193784 6866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:36.193830 6866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:36.193867 6866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:36.193917 6866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:36.202271 6866 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:35469
I20260928 23:49:36.202291 6994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:35469 every 8 connection(s)
I20260928 23:49:36.203197 6866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:49:36.209271 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 6866
I20260928 23:49:36.209360 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
I20260928 23:49:36.209609 6752 external_mini_cluster.cc:1428] Setting key b7f5bb24e39a5fc0a1aac74d1e7a4540
Sep 28 23:49:36 dist-test-slave-p5bw krb5kdc[6759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639376, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:36.217622 6842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:36.204936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:50119 (local address 127.6.152.62:38601)
0928 23:49:36.205076 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:36.205080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:36.205853 (+ 773us) server_negotiation.cc:407] Connection header received
0928 23:49:36.207175 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:36.207178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:36.207233 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:36.207342 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:36.208441 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:36.209076 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:36.210416 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:36.210735 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:36.210848 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:36.213716 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:36.213719 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:36.213721 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:36.213743 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:36.215428 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:36.216072 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:36.216075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:36.216077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:36.216196 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:36.216548 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:36.216551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:36.216553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:36.216726 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:36.216861 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:36.217310 (+ 449us) server_negotiation.cc:299] Negotiation successful
0928 23:49:36.217450 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:49:36.218358 6997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:36.205282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38601 (local address 127.6.152.1:50119)
0928 23:49:36.205712 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:36.205754 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:36.206972 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:36.207521 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:36.207527 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:36.207851 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:36.208299 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:36.208311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:36.209211 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:36.209214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:36.210313 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:36.210323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:36.210932 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:36.210937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:36.211138 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:36.211704 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:36.211729 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:36.213549 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:36.215552 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:36.215559 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:36.215573 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:36.215971 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:36.216299 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:36.216302 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:36.216304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:36.216446 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:36.216848 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:36.216854 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:36.217085 (+ 231us) client_negotiation.cc:769] Sending connection context
0928 23:49:36.217269 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 23:49:36.217583 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.6.152.2@KRBTEST.COM; defaulting to no policy
I20260928 23:49:36.219616 6995 heartbeater.cc:347] Connected to a master server at 127.6.152.62:38601
I20260928 23:49:36.219903 6995 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:36.220608 6995 heartbeater.cc:511] Master 127.6.152.62:38601 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.2@KRBTEST.COM" created.
I20260928 23:49:36.222311 6804 ts_manager.cc:194] Registered new tserver with Master: 72d86dae22154b119264b5dac2cf21cf (127.6.152.1:35469)
I20260928 23:49:36.223735 6804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.1@KRBTEST.COM'} at 127.6.152.1:50119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
I20260928 23:49:36.237802 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:38601
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:36.348564 7000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:36.348879 7000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:36.348995 7000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:36.353034 7000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:36.353103 7000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:36.353191 7000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:49:36.358487 7000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:38601
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:36.359462 7000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:36.360433 7000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:36.369971 7005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:36.369998 7006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:36.369999 7008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:36.370265 7000 server_base.cc:1034] running on GCE node
I20260928 23:49:36.370631 7000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:36.371203 7000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:36.372355 7000 hybrid_clock.cc:648] HybridClock initialized: now 1790639376372335 us; error 32 us; skew 500 ppm
Sep 28 23:49:36 dist-test-slave-p5bw krb5kdc[6759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639376, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:36.375131 7000 init.cc:377] Logged in from keytab as kudu/127.6.152.2@KRBTEST.COM (short username kudu)
I20260928 23:49:36.375990 7000 webserver.cc:460] Webserver started at http://127.6.152.2:39981/ using document root <none> and password file <none>
I20260928 23:49:36.376483 7000 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:36.376537 7000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:36.376690 7000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:36.378633 7000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "3af579be9d9349a9b24b8fef74982142"
format_stamp: "Formatted at 2026-09-28 23:49:36 on dist-test-slave-p5bw"
server_key: "4688c0ea2ccc266ca28402278b0ab999"
server_key_iv: "5b12e32daefae04a726d42c451d6e9d4"
server_key_version: "encryptionkey@0"
I20260928 23:49:36.379117 7000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "3af579be9d9349a9b24b8fef74982142"
format_stamp: "Formatted at 2026-09-28 23:49:36 on dist-test-slave-p5bw"
server_key: "4688c0ea2ccc266ca28402278b0ab999"
server_key_iv: "5b12e32daefae04a726d42c451d6e9d4"
server_key_version: "encryptionkey@0"
I20260928 23:49:36.382642 7000 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 23:49:36.384913 7015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:36.386060 7000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:49:36.386148 7000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "3af579be9d9349a9b24b8fef74982142"
format_stamp: "Formatted at 2026-09-28 23:49:36 on dist-test-slave-p5bw"
server_key: "4688c0ea2ccc266ca28402278b0ab999"
server_key_iv: "5b12e32daefae04a726d42c451d6e9d4"
server_key_version: "encryptionkey@0"
I20260928 23:49:36.386226 7000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:36.429003 7000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:36.432286 7000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:36.432440 7000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:36.433125 7000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:49:36.433991 7000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:36.434039 7000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:36.434078 7000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:36.434093 7000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:36.443382 7000 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:34717
I20260928 23:49:36.443442 7128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:34717 every 8 connection(s)
I20260928 23:49:36.444310 7000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
Sep 28 23:49:36 dist-test-slave-p5bw krb5kdc[6759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639376, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:36.454005 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 7000
I20260928 23:49:36.454111 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:49:36.454355 6752 external_mini_cluster.cc:1428] Setting key 6ca2eac006e60c4688ae280da12093b3
I20260928 23:49:36.457999 6842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:36.446129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:44323 (local address 127.6.152.62:38601)
0928 23:49:36.446258 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:36.446261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:36.447040 (+ 779us) server_negotiation.cc:407] Connection header received
0928 23:49:36.448380 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:36.448384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:36.448441 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:36.448552 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:36.449779 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:36.450388 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:36.451584 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:36.451817 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:36.451937 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:36.454318 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:36.454321 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:36.454324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:36.454352 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:36.455977 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:36.456466 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:36.456472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:36.456474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:36.456556 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:36.456926 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:36.456930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:36.456932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:36.457107 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:36.457209 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:36.457700 (+ 491us) server_negotiation.cc:299] Negotiation successful
0928 23:49:36.457835 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:49:36.458768 7131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:36.446480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38601 (local address 127.6.152.2:44323)
0928 23:49:36.446898 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:36.446939 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:36.448167 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:36.448724 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:36.448731 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:36.449150 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:36.449627 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:36.449640 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:36.450548 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:36.450551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:36.451447 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:36.451456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:36.451976 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:36.451980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:36.452151 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:36.452665 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:36.452681 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:36.454122 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:36.456087 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:36.456093 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:36.456108 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:36.456367 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:36.456646 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:36.456650 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:36.456652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:36.456792 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:36.457218 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:36.457227 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:36.457464 (+ 237us) client_negotiation.cc:769] Sending connection context
0928 23:49:36.457647 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 23:49:36.457972 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":103,"threads_started":1}
I20260928 23:49:36.459945 7129 heartbeater.cc:347] Connected to a master server at 127.6.152.62:38601
I20260928 23:49:36.460192 7129 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:36.460692 7129 heartbeater.cc:511] Master 127.6.152.62:38601 requested a full tablet report, sending...
I20260928 23:49:36.461763 6804 ts_manager.cc:194] Registered new tserver with Master: 3af579be9d9349a9b24b8fef74982142 (127.6.152.2:34717)
I20260928 23:49:36.462364 6804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.2@KRBTEST.COM'} at 127.6.152.2:44323
WARNING: no policy specified for kudu/127.6.152.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
I20260928 23:49:36.479772 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:38601
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:36.589680 7134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:36.589958 7134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:36.590031 7134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:36.594077 7134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:36.594152 7134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:36.594240 7134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:49:36.599339 7134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:38601
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:36.600368 7134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:36.601425 7134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:36.611059 7139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:36.611054 7140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:36.611092 7142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:36.611361 7134 server_base.cc:1034] running on GCE node
I20260928 23:49:36.611804 7134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:36.612468 7134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:36.613647 7134 hybrid_clock.cc:648] HybridClock initialized: now 1790639376613633 us; error 31 us; skew 500 ppm
Sep 28 23:49:36 dist-test-slave-p5bw krb5kdc[6759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639376, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:36.616400 7134 init.cc:377] Logged in from keytab as kudu/127.6.152.3@KRBTEST.COM (short username kudu)
I20260928 23:49:36.617296 7134 webserver.cc:460] Webserver started at http://127.6.152.3:40391/ using document root <none> and password file <none>
I20260928 23:49:36.617803 7134 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:36.617858 7134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:36.618013 7134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:36.619925 7134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "b31c495f79744e3380211f513f0fa5e7"
format_stamp: "Formatted at 2026-09-28 23:49:36 on dist-test-slave-p5bw"
server_key: "c523df5e8d6726ba7fd513b86ee004c9"
server_key_iv: "89b8baccbf841bab76cdade8bfada067"
server_key_version: "encryptionkey@0"
I20260928 23:49:36.620419 7134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "b31c495f79744e3380211f513f0fa5e7"
format_stamp: "Formatted at 2026-09-28 23:49:36 on dist-test-slave-p5bw"
server_key: "c523df5e8d6726ba7fd513b86ee004c9"
server_key_iv: "89b8baccbf841bab76cdade8bfada067"
server_key_version: "encryptionkey@0"
I20260928 23:49:36.624046 7134 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 23:49:36.626626 7149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:36.628055 7134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:49:36.628181 7134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "b31c495f79744e3380211f513f0fa5e7"
format_stamp: "Formatted at 2026-09-28 23:49:36 on dist-test-slave-p5bw"
server_key: "c523df5e8d6726ba7fd513b86ee004c9"
server_key_iv: "89b8baccbf841bab76cdade8bfada067"
server_key_version: "encryptionkey@0"
I20260928 23:49:36.628291 7134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:36.642236 7134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:36.645689 7134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:36.645895 7134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:36.646492 7134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:49:36.647293 7134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:36.647341 7134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:36.647378 7134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:36.647393 7134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:36.656770 7134 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:34043
I20260928 23:49:36.656798 7262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:34043 every 8 connection(s)
I20260928 23:49:36.657671 7134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:49:36.665552 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 7134
I20260928 23:49:36.665676 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:49:36.665974 6752 external_mini_cluster.cc:1428] Setting key ef09f574a74d0c9055ff399244ca2ee3
Sep 28 23:49:36 dist-test-slave-p5bw krb5kdc[6759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639376, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:36.671885 6842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:36.659499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:36055 (local address 127.6.152.62:38601)
0928 23:49:36.659663 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:36.659667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:36.660483 (+ 816us) server_negotiation.cc:407] Connection header received
0928 23:49:36.661971 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:36.661974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:36.662033 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:36.662137 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:36.663301 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:36.663845 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:36.665039 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:36.665267 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:36.665397 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:36.668190 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:36.668193 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:36.668195 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:36.668220 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:36.669805 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:36.670371 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:36.670373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:36.670375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:36.670448 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:36.670763 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:36.670765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:36.670767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:36.670974 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:36.671053 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:36.671589 (+ 536us) server_negotiation.cc:299] Negotiation successful
0928 23:49:36.671712 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 23:49:36.672680 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:36.659854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38601 (local address 127.6.152.3:36055)
0928 23:49:36.660317 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:36.660358 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:36.661730 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:36.662305 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:36.662312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:36.662675 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:36.663154 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:36.663169 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:36.663978 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:36.663983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:36.664921 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:36.664930 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:36.665438 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:36.665442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:36.665632 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:36.666308 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:36.666323 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:36.668002 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:36.669977 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:36.669983 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:36.669995 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:36.670275 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:36.670532 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:36.670535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:36.670536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:36.670675 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:36.671074 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:36.671079 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:36.671325 (+ 246us) client_negotiation.cc:769] Sending connection context
0928 23:49:36.671502 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 23:49:36.671767 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":107,"threads_started":1}
I20260928 23:49:36.673934 7263 heartbeater.cc:347] Connected to a master server at 127.6.152.62:38601
I20260928 23:49:36.674208 7263 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:36.674739 7263 heartbeater.cc:511] Master 127.6.152.62:38601 requested a full tablet report, sending...
I20260928 23:49:36.675838 6804 ts_manager.cc:194] Registered new tserver with Master: b31c495f79744e3380211f513f0fa5e7 (127.6.152.3:34043)
I20260928 23:49:36.676471 6804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.3@KRBTEST.COM'} at 127.6.152.3:36055
I20260928 23:49:36.680888 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 23:49:36 dist-test-slave-p5bw krb5kdc[6759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639376, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 23:49:36 dist-test-slave-p5bw krb5kdc[6759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639376, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.152.1@KRBTEST.COM
I20260928 23:49:36.707716 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:36.697599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36582 (local address 127.6.152.1:35469)
0928 23:49:36.697907 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:36.697925 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:36.698022 (+ 97us) server_negotiation.cc:407] Connection header received
0928 23:49:36.698208 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:36.698214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:36.698418 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:36.698567 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:36.699237 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:36.700009 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:36.700848 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:36.701217 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:36.701335 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:36.703796 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:36.703804 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:36.703816 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:36.703838 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:36.705663 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:36.706148 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:36.706156 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:36.706158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:36.706311 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:36.706643 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:36.706647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:36.706649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:36.706922 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:36.707168 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:36.707350 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 23:49:36.707516 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":113,"threads_started":1}
W20260928 23:49:36.708693 6909 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36582
Sep 28 23:49:36 dist-test-slave-p5bw krb5kdc[6759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 23:49:36 dist-test-slave-p5bw krb5kdc[6759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.1@KRBTEST.COM
I20260928 23:49:36.734807 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:36.725809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36584 (local address 127.6.152.1:35469)
0928 23:49:36.726007 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:36.726013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:36.726168 (+ 155us) server_negotiation.cc:407] Connection header received
0928 23:49:36.726339 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:36.726343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:36.726395 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:36.726484 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:36.727284 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:36.727847 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:36.729235 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:36.729481 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:36.729564 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:36.731934 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:36.731937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:36.731939 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:36.731970 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:36.733337 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:36.733755 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:36.733757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:36.733758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:36.733830 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:36.734050 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:36.734052 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:36.734053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:36.734201 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:36.734296 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:36.734495 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 23:49:36.734586 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 23:49:36.735809 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 6866
I20260928 23:49:36.742508 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 7000
I20260928 23:49:36.755095 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 7134
I20260928 23:49:36.761384 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 6772
2026-09-28T23:49:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2209 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:49:36 dist-test-slave-p5bw krb5kdc[7277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:49:36 dist-test-slave-p5bw krb5kdc[7277](info): set up 2 sockets
Sep 28 23:49:36 dist-test-slave-p5bw krb5kdc[7277](info): commencing operation
krb5kdc: starting...
W20260928 23:49:37.805065 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:49:37 dist-test-slave-p5bw krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:49:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:49:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:49:37.921201 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:40643
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:37735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:40643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:38.029419 7290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:38.029690 7290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:38.029732 7290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:38.033756 7290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:49:38.033803 7290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:38.033829 7290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:38.033885 7290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:49:38.033912 7290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:49:38.039202 7290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:37735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:40643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:40643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:38.040197 7290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:38.041225 7290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:38.049187 7295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:38.049265 7290 server_base.cc:1034] running on GCE node
W20260928 23:49:38.049206 7296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:38.049202 7298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:38.049973 7290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:38.050993 7290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:38.052181 7290 hybrid_clock.cc:648] HybridClock initialized: now 1790639378052158 us; error 36 us; skew 500 ppm
Sep 28 23:49:38 dist-test-slave-p5bw krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639378, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:38.055876 7290 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:49:38.056740 7290 webserver.cc:460] Webserver started at http://127.6.152.62:36251/ using document root <none> and password file <none>
I20260928 23:49:38.057322 7290 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:38.057384 7290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:38.057605 7290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:38.059692 7290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "2626a1a496a443a8b716b0bdf3a4c657"
format_stamp: "Formatted at 2026-09-28 23:49:38 on dist-test-slave-p5bw"
server_key: "d8819b24e914bdb47da728aafdb3ccf8"
server_key_iv: "217b782e089320abceffc0beca2a8fd3"
server_key_version: "encryptionkey@0"
I20260928 23:49:38.060276 7290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "2626a1a496a443a8b716b0bdf3a4c657"
format_stamp: "Formatted at 2026-09-28 23:49:38 on dist-test-slave-p5bw"
server_key: "d8819b24e914bdb47da728aafdb3ccf8"
server_key_iv: "217b782e089320abceffc0beca2a8fd3"
server_key_version: "encryptionkey@0"
I20260928 23:49:38.064100 7290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:49:38.066632 7305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:38.067832 7290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:49:38.067966 7290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "2626a1a496a443a8b716b0bdf3a4c657"
format_stamp: "Formatted at 2026-09-28 23:49:38 on dist-test-slave-p5bw"
server_key: "d8819b24e914bdb47da728aafdb3ccf8"
server_key_iv: "217b782e089320abceffc0beca2a8fd3"
server_key_version: "encryptionkey@0"
I20260928 23:49:38.068073 7290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:38.082566 7290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:38.085500 7290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:38.085739 7290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:38.092437 7290 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:40643
I20260928 23:49:38.092495 7357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:40643 every 8 connection(s)
I20260928 23:49:38.093467 7290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:49:38.096266 7358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:38.096448 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 7290
I20260928 23:49:38.096570 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:49:38.096876 6752 external_mini_cluster.cc:1428] Setting key f2abb10ec33e979e578d0280d799e6d2
I20260928 23:49:38.102656 7358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2626a1a496a443a8b716b0bdf3a4c657: Bootstrap starting.
I20260928 23:49:38.104941 7358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2626a1a496a443a8b716b0bdf3a4c657: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:38.105688 7358 log.cc:824] T 00000000000000000000000000000000 P 2626a1a496a443a8b716b0bdf3a4c657: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:49:38 dist-test-slave-p5bw krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:38.107693 7358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2626a1a496a443a8b716b0bdf3a4c657: No bootstrap required, opened a new log
I20260928 23:49:38.111299 7361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:38.098298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56314 (local address 127.6.152.62:40643)
0928 23:49:38.098731 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:38.098746 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:38.098780 (+ 34us) server_negotiation.cc:407] Connection header received
0928 23:49:38.099565 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:38.099583 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:38.099906 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:38.100237 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:38.101066 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.102036 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.103002 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.103492 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.103670 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:38.106479 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:38.106486 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:38.106505 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:38.106538 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:38.108375 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:38.108946 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:38.108951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:38.108955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:38.109067 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:38.109358 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:38.109360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:38.109361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:38.109747 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:38.109921 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:38.110244 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 23:49:38.110550 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":133,"threads_started":1}
I20260928 23:49:38.111902 7358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2626a1a496a443a8b716b0bdf3a4c657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2626a1a496a443a8b716b0bdf3a4c657" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40643 } }
I20260928 23:49:38.112182 7358 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2626a1a496a443a8b716b0bdf3a4c657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:38.112237 7358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2626a1a496a443a8b716b0bdf3a4c657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2626a1a496a443a8b716b0bdf3a4c657, State: Initialized, Role: FOLLOWER
I20260928 23:49:38.112700 7358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2626a1a496a443a8b716b0bdf3a4c657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2626a1a496a443a8b716b0bdf3a4c657" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40643 } }
I20260928 23:49:38.112890 7358 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2626a1a496a443a8b716b0bdf3a4c657 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:49:38.112967 7358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2626a1a496a443a8b716b0bdf3a4c657 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:49:38.113085 7358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2626a1a496a443a8b716b0bdf3a4c657 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:38.114161 7358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2626a1a496a443a8b716b0bdf3a4c657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2626a1a496a443a8b716b0bdf3a4c657" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40643 } }
I20260928 23:49:38.114503 7358 leader_election.cc:302] T 00000000000000000000000000000000 P 2626a1a496a443a8b716b0bdf3a4c657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2626a1a496a443a8b716b0bdf3a4c657; no voters:
I20260928 23:49:38.114941 7358 leader_election.cc:288] T 00000000000000000000000000000000 P 2626a1a496a443a8b716b0bdf3a4c657 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:49:38.115314 7364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2626a1a496a443a8b716b0bdf3a4c657 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:49:38.115538 7364 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2626a1a496a443a8b716b0bdf3a4c657 [term 1 LEADER]: Becoming Leader. State: Replica: 2626a1a496a443a8b716b0bdf3a4c657, State: Running, Role: LEADER
I20260928 23:49:38.115890 7364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2626a1a496a443a8b716b0bdf3a4c657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2626a1a496a443a8b716b0bdf3a4c657" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40643 } }
I20260928 23:49:38.116284 7358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2626a1a496a443a8b716b0bdf3a4c657 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:49:38.117794 7363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2626a1a496a443a8b716b0bdf3a4c657 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2626a1a496a443a8b716b0bdf3a4c657. Latest consensus state: current_term: 1 leader_uuid: "2626a1a496a443a8b716b0bdf3a4c657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2626a1a496a443a8b716b0bdf3a4c657" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40643 } } }
I20260928 23:49:38.117925 7363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2626a1a496a443a8b716b0bdf3a4c657 [sys.catalog]: This master's current role is: LEADER
I20260928 23:49:38.117831 7365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2626a1a496a443a8b716b0bdf3a4c657 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2626a1a496a443a8b716b0bdf3a4c657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2626a1a496a443a8b716b0bdf3a4c657" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40643 } } }
I20260928 23:49:38.117978 7365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2626a1a496a443a8b716b0bdf3a4c657 [sys.catalog]: This master's current role is: LEADER
I20260928 23:49:38.118312 7374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:49:38.121618 7374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:49:38.127288 7374 catalog_manager.cc:1347] Generated new cluster ID: 932c730927d84e22b16f442df557092e
I20260928 23:49:38.127365 7374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:49:38.143383 7374 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:49:38.144414 7374 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:49:38.155381 7374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2626a1a496a443a8b716b0bdf3a4c657: Generated new TSK 0
I20260928 23:49:38.156102 7374 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
I20260928 23:49:38.191442 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:40643
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:37735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:38.299865 7384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:38.300099 7384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:38.300141 7384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:38.304093 7384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:38.304148 7384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:38.304235 7384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:49:38.309484 7384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:37735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:40643
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:38.310465 7384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:38.311455 7384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:38.320726 7390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:38.320794 7392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:38.320904 7389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:38.320945 7384 server_base.cc:1034] running on GCE node
I20260928 23:49:38.321378 7384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:38.321974 7384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:38.323158 7384 hybrid_clock.cc:648] HybridClock initialized: now 1790639378323124 us; error 46 us; skew 500 ppm
Sep 28 23:49:38 dist-test-slave-p5bw krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639378, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:38.325850 7384 init.cc:377] Logged in from keytab as kudu/127.6.152.1@KRBTEST.COM (short username kudu)
I20260928 23:49:38.326812 7384 webserver.cc:460] Webserver started at http://127.6.152.1:32915/ using document root <none> and password file <none>
I20260928 23:49:38.327379 7384 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:38.327484 7384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:38.327711 7384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:38.329806 7384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "2053e34c75ff4bc089e392c703f9f2f6"
format_stamp: "Formatted at 2026-09-28 23:49:38 on dist-test-slave-p5bw"
server_key: "d2aa79399dfb0169e77a4d3755fc0577"
server_key_iv: "713dc989bbd8fdfe358e2030a495564b"
server_key_version: "encryptionkey@0"
I20260928 23:49:38.330377 7384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "2053e34c75ff4bc089e392c703f9f2f6"
format_stamp: "Formatted at 2026-09-28 23:49:38 on dist-test-slave-p5bw"
server_key: "d2aa79399dfb0169e77a4d3755fc0577"
server_key_iv: "713dc989bbd8fdfe358e2030a495564b"
server_key_version: "encryptionkey@0"
I20260928 23:49:38.333993 7384 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:49:38.336481 7399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:38.337926 7384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:49:38.338073 7384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "2053e34c75ff4bc089e392c703f9f2f6"
format_stamp: "Formatted at 2026-09-28 23:49:38 on dist-test-slave-p5bw"
server_key: "d2aa79399dfb0169e77a4d3755fc0577"
server_key_iv: "713dc989bbd8fdfe358e2030a495564b"
server_key_version: "encryptionkey@0"
I20260928 23:49:38.338184 7384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:38.361045 7384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:38.364378 7384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:38.364591 7384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:38.365238 7384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:49:38.366035 7384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:38.366079 7384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:38.366148 7384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:38.366178 7384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:38.374730 7384 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:34617
I20260928 23:49:38.374760 7512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:34617 every 8 connection(s)
I20260928 23:49:38.375599 7384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:49:38.377226 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 7384
I20260928 23:49:38.377331 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
I20260928 23:49:38.377630 6752 external_mini_cluster.cc:1428] Setting key f8805313b7d12b43cd50671d7fd62f5d
Sep 28 23:49:38 dist-test-slave-p5bw krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639378, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.152.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.2@KRBTEST.COM" created.
I20260928 23:49:38.390746 7361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:38.377447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:51107 (local address 127.6.152.62:40643)
0928 23:49:38.377567 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:38.377570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:38.378649 (+ 1079us) server_negotiation.cc:407] Connection header received
0928 23:49:38.380069 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:38.380072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:38.380125 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:38.380227 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:38.381830 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.382375 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.384229 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.384504 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.384585 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:38.387174 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:38.387178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:38.387179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:38.387202 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:38.388428 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:38.389000 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:38.389003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:38.389007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:38.389082 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:38.389486 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:38.389508 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:38.389513 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:38.389754 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:38.389864 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:38.390357 (+ 493us) server_negotiation.cc:299] Negotiation successful
0928 23:49:38.390511 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:49:38.391589 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:38.377929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40643 (local address 127.6.152.1:51107)
0928 23:49:38.378518 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:38.378560 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:38.379840 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:38.380418 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:38.380430 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:38.380981 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:38.381657 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:38.381678 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.382502 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.382508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:38.384104 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:38.384118 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.384661 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.384668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:38.384960 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:38.385610 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:38.385626 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:38.387019 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:38.388558 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:38.388564 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:38.388577 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:38.388876 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:38.389180 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:38.389183 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:38.389185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:38.389361 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:38.389895 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:38.389900 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:38.390112 (+ 212us) client_negotiation.cc:769] Sending connection context
0928 23:49:38.390259 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 23:49:38.390490 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":159,"threads_started":1}
I20260928 23:49:38.392925 7513 heartbeater.cc:347] Connected to a master server at 127.6.152.62:40643
I20260928 23:49:38.393167 7513 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:38.393674 7513 heartbeater.cc:511] Master 127.6.152.62:40643 requested a full tablet report, sending...
I20260928 23:49:38.395285 7322 ts_manager.cc:194] Registered new tserver with Master: 2053e34c75ff4bc089e392c703f9f2f6 (127.6.152.1:34617)
I20260928 23:49:38.397017 7322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.1@KRBTEST.COM'} at 127.6.152.1:51107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
I20260928 23:49:38.403676 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:40643
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:37735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:38.510588 7518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:38.510840 7518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:38.510883 7518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:38.514779 7518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:38.514827 7518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:38.514883 7518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:49:38.520305 7518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:37735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:40643
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:38.521322 7518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:38.522280 7518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:38.531638 7524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:38.531638 7523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:38.531692 7518 server_base.cc:1034] running on GCE node
W20260928 23:49:38.531896 7526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:38.532366 7518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:38.532977 7518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:38.534132 7518 hybrid_clock.cc:648] HybridClock initialized: now 1790639378534120 us; error 23 us; skew 500 ppm
Sep 28 23:49:38 dist-test-slave-p5bw krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639378, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:38.536609 7518 init.cc:377] Logged in from keytab as kudu/127.6.152.2@KRBTEST.COM (short username kudu)
I20260928 23:49:38.537586 7518 webserver.cc:460] Webserver started at http://127.6.152.2:35891/ using document root <none> and password file <none>
I20260928 23:49:38.538168 7518 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:38.538260 7518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:38.538457 7518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:38.540416 7518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "8bb32ac1678f48d0bacca044aac7d05f"
format_stamp: "Formatted at 2026-09-28 23:49:38 on dist-test-slave-p5bw"
server_key: "3b92e6c7304889ed3f0fa64ebbb8f67b"
server_key_iv: "0ad7be3f9701f3929694ed48c46523e0"
server_key_version: "encryptionkey@0"
I20260928 23:49:38.541039 7518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "8bb32ac1678f48d0bacca044aac7d05f"
format_stamp: "Formatted at 2026-09-28 23:49:38 on dist-test-slave-p5bw"
server_key: "3b92e6c7304889ed3f0fa64ebbb8f67b"
server_key_iv: "0ad7be3f9701f3929694ed48c46523e0"
server_key_version: "encryptionkey@0"
I20260928 23:49:38.544588 7518 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 23:49:38.546845 7533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:38.548005 7518 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:49:38.548132 7518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "8bb32ac1678f48d0bacca044aac7d05f"
format_stamp: "Formatted at 2026-09-28 23:49:38 on dist-test-slave-p5bw"
server_key: "3b92e6c7304889ed3f0fa64ebbb8f67b"
server_key_iv: "0ad7be3f9701f3929694ed48c46523e0"
server_key_version: "encryptionkey@0"
I20260928 23:49:38.548239 7518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:38.580552 7518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:38.583519 7518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:38.583724 7518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:38.584308 7518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:49:38.585189 7518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:38.585259 7518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:38.585330 7518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:38.585378 7518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:38.594388 7518 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:35707
I20260928 23:49:38.594425 7646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:35707 every 8 connection(s)
I20260928 23:49:38.595278 7518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:49:38.599398 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 7518
I20260928 23:49:38.599534 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:49:38.599797 6752 external_mini_cluster.cc:1428] Setting key 11b8cced1a62a3c715258c649192dc51
Sep 28 23:49:38 dist-test-slave-p5bw krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639378, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.152.3@KRBTEST.COM; defaulting to no policy
I20260928 23:49:38.609462 7361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:38.597055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:57125 (local address 127.6.152.62:40643)
0928 23:49:38.597204 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:38.597208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:38.598057 (+ 849us) server_negotiation.cc:407] Connection header received
0928 23:49:38.599346 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:38.599350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:38.599408 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:38.599522 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:38.600939 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.601695 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.602880 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.603103 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.603213 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:38.605899 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:38.605902 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:38.605904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:38.605927 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:38.607362 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:38.607875 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:38.607880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:38.607882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:38.607970 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:38.608301 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:38.608306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:38.608308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:38.608488 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:38.608595 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:38.609191 (+ 596us) server_negotiation.cc:299] Negotiation successful
0928 23:49:38.609304 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:49:38.610322 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:38.597488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40643 (local address 127.6.152.2:57125)
0928 23:49:38.597907 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:38.597949 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:38.599114 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:38.599698 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:38.599705 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:38.600130 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:38.600783 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:38.600803 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.601821 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.601823 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:38.602746 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:38.602755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.603247 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.603251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:38.603431 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:38.604043 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:38.604067 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:38.605724 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:38.607499 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:38.607505 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:38.607520 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:38.607771 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:38.608064 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:38.608068 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:38.608070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:38.608207 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:38.608593 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:38.608601 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:38.608907 (+ 306us) client_negotiation.cc:769] Sending connection context
0928 23:49:38.609147 (+ 240us) client_negotiation.cc:240] Negotiation successful
0928 23:49:38.609456 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.3@KRBTEST.COM" created.
I20260928 23:49:38.611624 7647 heartbeater.cc:347] Connected to a master server at 127.6.152.62:40643
I20260928 23:49:38.611867 7647 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:38.612346 7647 heartbeater.cc:511] Master 127.6.152.62:40643 requested a full tablet report, sending...
I20260928 23:49:38.613405 7322 ts_manager.cc:194] Registered new tserver with Master: 8bb32ac1678f48d0bacca044aac7d05f (127.6.152.2:35707)
I20260928 23:49:38.614274 7322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.2@KRBTEST.COM'} at 127.6.152.2:57125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
I20260928 23:49:38.627158 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:40643
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:37735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:38.732673 7652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:38.732962 7652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:38.733008 7652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:38.736792 7652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:38.736871 7652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:38.736927 7652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:49:38.742066 7652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:37735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:40643
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:38.743043 7652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:38.743923 7652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:38.752892 7660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:38.752938 7657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:38.752909 7658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:38.753108 7652 server_base.cc:1034] running on GCE node
I20260928 23:49:38.753571 7652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:38.754129 7652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:38.755282 7652 hybrid_clock.cc:648] HybridClock initialized: now 1790639378755265 us; error 28 us; skew 500 ppm
Sep 28 23:49:38 dist-test-slave-p5bw krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639378, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:38.757831 7652 init.cc:377] Logged in from keytab as kudu/127.6.152.3@KRBTEST.COM (short username kudu)
I20260928 23:49:38.758659 7652 webserver.cc:460] Webserver started at http://127.6.152.3:37821/ using document root <none> and password file <none>
I20260928 23:49:38.759183 7652 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:38.759239 7652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:38.759469 7652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:38.761411 7652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "22a7fe63093d410a9ba1a18b57c49f9d"
format_stamp: "Formatted at 2026-09-28 23:49:38 on dist-test-slave-p5bw"
server_key: "9cedef597d850b2faf2faa969cf4809c"
server_key_iv: "9284b78c00f7c14490294a1a313dcfcf"
server_key_version: "encryptionkey@0"
I20260928 23:49:38.761927 7652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "22a7fe63093d410a9ba1a18b57c49f9d"
format_stamp: "Formatted at 2026-09-28 23:49:38 on dist-test-slave-p5bw"
server_key: "9cedef597d850b2faf2faa969cf4809c"
server_key_iv: "9284b78c00f7c14490294a1a313dcfcf"
server_key_version: "encryptionkey@0"
I20260928 23:49:38.765305 7652 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:49:38.767434 7667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:38.768575 7652 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:49:38.768699 7652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "22a7fe63093d410a9ba1a18b57c49f9d"
format_stamp: "Formatted at 2026-09-28 23:49:38 on dist-test-slave-p5bw"
server_key: "9cedef597d850b2faf2faa969cf4809c"
server_key_iv: "9284b78c00f7c14490294a1a313dcfcf"
server_key_version: "encryptionkey@0"
I20260928 23:49:38.768802 7652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:38.791982 7652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:38.794835 7652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:38.795055 7652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:38.795620 7652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:49:38.796406 7652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:38.796475 7652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:38.796547 7652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:38.796597 7652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:38.805305 7652 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:33721
I20260928 23:49:38.805337 7780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:33721 every 8 connection(s)
I20260928 23:49:38.806137 7652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:49:38.813066 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 7652
I20260928 23:49:38.813187 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:49:38.813465 6752 external_mini_cluster.cc:1428] Setting key b6c7c57357af2105850580bcb6deaab6
Sep 28 23:49:38 dist-test-slave-p5bw krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639378, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:38.819281 7361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:38.807830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:40191 (local address 127.6.152.62:40643)
0928 23:49:38.807966 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:38.807970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:38.808781 (+ 811us) server_negotiation.cc:407] Connection header received
0928 23:49:38.810091 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:38.810094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:38.810138 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:38.810211 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:38.811305 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.811816 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.812940 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.813192 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.813323 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:38.816020 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:38.816023 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:38.816024 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:38.816047 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:38.817565 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:38.818000 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:38.818003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:38.818004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:38.818080 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:38.818416 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:38.818419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:38.818420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:38.818588 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:38.818664 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:38.819075 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 23:49:38.819168 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:49:38.820062 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:38.808208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40643 (local address 127.6.152.3:40191)
0928 23:49:38.808629 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:38.808669 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:38.809889 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:38.810366 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:38.810372 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:38.810703 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:38.811177 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:38.811191 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.811913 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.811915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:38.812769 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:38.812777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.813362 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.813366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:38.813565 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:38.814201 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:38.814220 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:38.815879 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:38.817661 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:38.817665 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:38.817676 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:38.817919 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:38.818193 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:38.818196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:38.818198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:38.818328 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:38.818673 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:38.818677 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:38.818887 (+ 210us) client_negotiation.cc:769] Sending connection context
0928 23:49:38.819031 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 23:49:38.819257 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":110,"threads_started":1}
I20260928 23:49:38.821086 7781 heartbeater.cc:347] Connected to a master server at 127.6.152.62:40643
I20260928 23:49:38.821321 7781 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:38.821761 7781 heartbeater.cc:511] Master 127.6.152.62:40643 requested a full tablet report, sending...
I20260928 23:49:38.822667 7322 ts_manager.cc:194] Registered new tserver with Master: 22a7fe63093d410a9ba1a18b57c49f9d (127.6.152.3:33721)
I20260928 23:49:38.823240 7322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.3@KRBTEST.COM'} at 127.6.152.3:40191
I20260928 23:49:38.827025 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:49:38.837054 7361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:38.830352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56318 (local address 127.6.152.62:40643)
0928 23:49:38.830480 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:38.830485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:38.830638 (+ 153us) server_negotiation.cc:407] Connection header received
0928 23:49:38.830733 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:38.830736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:38.830770 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:38.830833 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:38.831479 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.831954 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.832716 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.832981 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.833059 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:38.834233 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:38.834236 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:38.834238 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:38.834259 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:38.835624 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:38.836059 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:38.836061 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:38.836063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:38.836131 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:38.836333 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:38.836335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:38.836336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:38.836517 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:38.836612 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:38.836772 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 23:49:38.836909 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:49:38.841737 7322 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:49:38.844746 7322 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:49:38.858299 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:38.852573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:34617 (local address 127.0.0.1:50968)
0928 23:49:38.852950 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:38.853000 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:38.853122 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:38.853865 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:38.853871 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:38.853896 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:49:38.854020 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:38.854027 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.855372 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.855375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:38.856679 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:38.856688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.857733 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.857737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:38.857887 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:38.857931 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 23:49:38.858040 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 23:49:38.858148 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":4,"thread_start_us":82,"threads_started":1}
I20260928 23:49:38.858336 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:38.852725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50968 (local address 127.6.152.1:34617)
0928 23:49:38.853038 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:38.853055 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:38.853080 (+ 25us) server_negotiation.cc:407] Connection header received
0928 23:49:38.853277 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:38.853284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:38.853515 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:38.853667 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:49:38.854171 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.855238 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.856786 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.857626 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.857729 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:38.857884 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:38.858087 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 23:49:38.858181 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":133,"threads_started":1}
I20260928 23:49:38.859404 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:38.853313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:33721 (local address 127.0.0.1:52702)
0928 23:49:38.854246 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:38.854258 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:38.854382 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:38.854823 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:38.854825 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:38.854841 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:49:38.854932 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:38.854938 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.856320 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.856323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:38.857545 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:38.857557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.859007 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.859011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:38.859095 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:38.859118 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:49:38.859202 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 23:49:38.859271 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"mutex_wait_us":16,"thread_start_us":93,"threads_started":1}
I20260928 23:49:38.859622 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:38.853531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52702 (local address 127.6.152.3:33721)
0928 23:49:38.853960 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:38.853974 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:38.854356 (+ 382us) server_negotiation.cc:407] Connection header received
0928 23:49:38.854487 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:38.854493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:38.854678 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:38.854844 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:49:38.855026 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.856176 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.857666 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.858886 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.858990 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:38.859116 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:38.859224 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 23:49:38.859304 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":123,"threads_started":1}
I20260928 23:49:38.860368 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:38.853630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:35707 (local address 127.0.0.1:34196)
0928 23:49:38.854552 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:38.854563 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:38.854638 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:38.855057 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:38.855062 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:38.855080 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:49:38.855157 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:38.855162 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.856232 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.856236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:38.858680 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:38.858692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.859967 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.859971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:38.860056 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:38.860084 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 23:49:38.860161 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 23:49:38.860224 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"mutex_wait_us":53,"thread_start_us":87,"threads_started":1}
I20260928 23:49:38.860724 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:38.854003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34196 (local address 127.6.152.2:35707)
0928 23:49:38.854298 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:38.854309 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:38.854647 (+ 338us) server_negotiation.cc:407] Connection header received
0928 23:49:38.854726 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:38.854731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:38.854900 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:38.855048 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:49:38.855266 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.856082 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.858812 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.859661 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.859770 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:38.859905 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:38.860213 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 23:49:38.860308 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":94,"threads_started":1}
I20260928 23:49:38.861595 7447 tablet_service.cc:1467] Processing CreateTablet for tablet 57d403ee7e344e1591aeb1312bc9904f (DEFAULT_TABLE table=test-table [id=37e2fa3f7dff4b14b24a8e303d731c69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:38.861935 7715 tablet_service.cc:1467] Processing CreateTablet for tablet 57d403ee7e344e1591aeb1312bc9904f (DEFAULT_TABLE table=test-table [id=37e2fa3f7dff4b14b24a8e303d731c69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:38.862543 7447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57d403ee7e344e1591aeb1312bc9904f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:38.862955 7715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57d403ee7e344e1591aeb1312bc9904f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:38.864542 7581 tablet_service.cc:1467] Processing CreateTablet for tablet 57d403ee7e344e1591aeb1312bc9904f (DEFAULT_TABLE table=test-table [id=37e2fa3f7dff4b14b24a8e303d731c69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:38.865545 7581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57d403ee7e344e1591aeb1312bc9904f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:38.868911 7798 tablet_bootstrap.cc:492] T 57d403ee7e344e1591aeb1312bc9904f P 22a7fe63093d410a9ba1a18b57c49f9d: Bootstrap starting.
I20260928 23:49:38.870265 7799 tablet_bootstrap.cc:492] T 57d403ee7e344e1591aeb1312bc9904f P 2053e34c75ff4bc089e392c703f9f2f6: Bootstrap starting.
I20260928 23:49:38.871773 7798 tablet_bootstrap.cc:654] T 57d403ee7e344e1591aeb1312bc9904f P 22a7fe63093d410a9ba1a18b57c49f9d: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:38.872599 7799 tablet_bootstrap.cc:654] T 57d403ee7e344e1591aeb1312bc9904f P 2053e34c75ff4bc089e392c703f9f2f6: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:38.872699 7798 log.cc:824] T 57d403ee7e344e1591aeb1312bc9904f P 22a7fe63093d410a9ba1a18b57c49f9d: Log is configured to *not* fsync() on all Append() calls
I20260928 23:49:38.873313 7800 tablet_bootstrap.cc:492] T 57d403ee7e344e1591aeb1312bc9904f P 8bb32ac1678f48d0bacca044aac7d05f: Bootstrap starting.
I20260928 23:49:38.873895 7799 log.cc:824] T 57d403ee7e344e1591aeb1312bc9904f P 2053e34c75ff4bc089e392c703f9f2f6: Log is configured to *not* fsync() on all Append() calls
I20260928 23:49:38.874766 7798 tablet_bootstrap.cc:492] T 57d403ee7e344e1591aeb1312bc9904f P 22a7fe63093d410a9ba1a18b57c49f9d: No bootstrap required, opened a new log
I20260928 23:49:38.874953 7798 ts_tablet_manager.cc:1397] T 57d403ee7e344e1591aeb1312bc9904f P 22a7fe63093d410a9ba1a18b57c49f9d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 23:49:38.875772 7800 tablet_bootstrap.cc:654] T 57d403ee7e344e1591aeb1312bc9904f P 8bb32ac1678f48d0bacca044aac7d05f: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:38.875808 7799 tablet_bootstrap.cc:492] T 57d403ee7e344e1591aeb1312bc9904f P 2053e34c75ff4bc089e392c703f9f2f6: No bootstrap required, opened a new log
I20260928 23:49:38.875985 7799 ts_tablet_manager.cc:1397] T 57d403ee7e344e1591aeb1312bc9904f P 2053e34c75ff4bc089e392c703f9f2f6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:49:38.876453 7800 log.cc:824] T 57d403ee7e344e1591aeb1312bc9904f P 8bb32ac1678f48d0bacca044aac7d05f: Log is configured to *not* fsync() on all Append() calls
I20260928 23:49:38.878278 7800 tablet_bootstrap.cc:492] T 57d403ee7e344e1591aeb1312bc9904f P 8bb32ac1678f48d0bacca044aac7d05f: No bootstrap required, opened a new log
I20260928 23:49:38.878475 7800 ts_tablet_manager.cc:1397] T 57d403ee7e344e1591aeb1312bc9904f P 8bb32ac1678f48d0bacca044aac7d05f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 23:49:38.878834 7798 raft_consensus.cc:348] T 57d403ee7e344e1591aeb1312bc9904f P 22a7fe63093d410a9ba1a18b57c49f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2053e34c75ff4bc089e392c703f9f2f6" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34617 } } peers { permanent_uuid: "22a7fe63093d410a9ba1a18b57c49f9d" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33721 } } peers { permanent_uuid: "8bb32ac1678f48d0bacca044aac7d05f" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 35707 } }
I20260928 23:49:38.879110 7798 raft_consensus.cc:374] T 57d403ee7e344e1591aeb1312bc9904f P 22a7fe63093d410a9ba1a18b57c49f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:38.879168 7798 raft_consensus.cc:729] T 57d403ee7e344e1591aeb1312bc9904f P 22a7fe63093d410a9ba1a18b57c49f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22a7fe63093d410a9ba1a18b57c49f9d, State: Initialized, Role: FOLLOWER
I20260928 23:49:38.879612 7798 consensus_queue.cc:260] T 57d403ee7e344e1591aeb1312bc9904f P 22a7fe63093d410a9ba1a18b57c49f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2053e34c75ff4bc089e392c703f9f2f6" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34617 } } peers { permanent_uuid: "22a7fe63093d410a9ba1a18b57c49f9d" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33721 } } peers { permanent_uuid: "8bb32ac1678f48d0bacca044aac7d05f" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 35707 } }
I20260928 23:49:38.879666 7799 raft_consensus.cc:348] T 57d403ee7e344e1591aeb1312bc9904f P 2053e34c75ff4bc089e392c703f9f2f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2053e34c75ff4bc089e392c703f9f2f6" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34617 } } peers { permanent_uuid: "22a7fe63093d410a9ba1a18b57c49f9d" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33721 } } peers { permanent_uuid: "8bb32ac1678f48d0bacca044aac7d05f" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 35707 } }
I20260928 23:49:38.879915 7799 raft_consensus.cc:374] T 57d403ee7e344e1591aeb1312bc9904f P 2053e34c75ff4bc089e392c703f9f2f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:38.879974 7799 raft_consensus.cc:729] T 57d403ee7e344e1591aeb1312bc9904f P 2053e34c75ff4bc089e392c703f9f2f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2053e34c75ff4bc089e392c703f9f2f6, State: Initialized, Role: FOLLOWER
I20260928 23:49:38.880352 7781 heartbeater.cc:503] Master 127.6.152.62:40643 was elected leader, sending a full tablet report...
I20260928 23:49:38.880393 7799 consensus_queue.cc:260] T 57d403ee7e344e1591aeb1312bc9904f P 2053e34c75ff4bc089e392c703f9f2f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2053e34c75ff4bc089e392c703f9f2f6" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34617 } } peers { permanent_uuid: "22a7fe63093d410a9ba1a18b57c49f9d" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33721 } } peers { permanent_uuid: "8bb32ac1678f48d0bacca044aac7d05f" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 35707 } }
I20260928 23:49:38.880607 7798 ts_tablet_manager.cc:1428] T 57d403ee7e344e1591aeb1312bc9904f P 22a7fe63093d410a9ba1a18b57c49f9d: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260928 23:49:38.881201 7513 heartbeater.cc:503] Master 127.6.152.62:40643 was elected leader, sending a full tablet report...
I20260928 23:49:38.881268 7799 ts_tablet_manager.cc:1428] T 57d403ee7e344e1591aeb1312bc9904f P 2053e34c75ff4bc089e392c703f9f2f6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 23:49:38.881844 7800 raft_consensus.cc:348] T 57d403ee7e344e1591aeb1312bc9904f P 8bb32ac1678f48d0bacca044aac7d05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2053e34c75ff4bc089e392c703f9f2f6" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34617 } } peers { permanent_uuid: "22a7fe63093d410a9ba1a18b57c49f9d" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33721 } } peers { permanent_uuid: "8bb32ac1678f48d0bacca044aac7d05f" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 35707 } }
I20260928 23:49:38.882093 7800 raft_consensus.cc:374] T 57d403ee7e344e1591aeb1312bc9904f P 8bb32ac1678f48d0bacca044aac7d05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:38.882156 7800 raft_consensus.cc:729] T 57d403ee7e344e1591aeb1312bc9904f P 8bb32ac1678f48d0bacca044aac7d05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bb32ac1678f48d0bacca044aac7d05f, State: Initialized, Role: FOLLOWER
I20260928 23:49:38.882547 7800 consensus_queue.cc:260] T 57d403ee7e344e1591aeb1312bc9904f P 8bb32ac1678f48d0bacca044aac7d05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2053e34c75ff4bc089e392c703f9f2f6" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34617 } } peers { permanent_uuid: "22a7fe63093d410a9ba1a18b57c49f9d" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33721 } } peers { permanent_uuid: "8bb32ac1678f48d0bacca044aac7d05f" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 35707 } }
I20260928 23:49:38.883293 7647 heartbeater.cc:503] Master 127.6.152.62:40643 was elected leader, sending a full tablet report...
I20260928 23:49:38.884410 7800 ts_tablet_manager.cc:1428] T 57d403ee7e344e1591aeb1312bc9904f P 8bb32ac1678f48d0bacca044aac7d05f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:49:38.912235 7806 raft_consensus.cc:482] T 57d403ee7e344e1591aeb1312bc9904f P 8bb32ac1678f48d0bacca044aac7d05f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:49:38.912464 7806 raft_consensus.cc:504] T 57d403ee7e344e1591aeb1312bc9904f P 8bb32ac1678f48d0bacca044aac7d05f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2053e34c75ff4bc089e392c703f9f2f6" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34617 } } peers { permanent_uuid: "22a7fe63093d410a9ba1a18b57c49f9d" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33721 } } peers { permanent_uuid: "8bb32ac1678f48d0bacca044aac7d05f" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 35707 } }
I20260928 23:49:38.913616 7806 leader_election.cc:288] T 57d403ee7e344e1591aeb1312bc9904f P 8bb32ac1678f48d0bacca044aac7d05f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2053e34c75ff4bc089e392c703f9f2f6 (127.6.152.1:34617), 22a7fe63093d410a9ba1a18b57c49f9d (127.6.152.3:33721)
I20260928 23:49:38.917667 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:38.913882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:34617 (local address 127.6.152.2:59011)
0928 23:49:38.914013 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:38.914028 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:38.914116 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:38.914380 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:38.914383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:38.914400 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:49:38.914520 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:38.914531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.915264 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.915269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:38.916423 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:38.916437 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.917196 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.917199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:38.917347 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:38.917371 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 23:49:38.917439 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 23:49:38.917536 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 23:49:38.917666 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:38.914003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:59011 (local address 127.6.152.1:34617)
0928 23:49:38.914126 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:38.914132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:38.914148 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:49:38.914204 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:38.914207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:38.914260 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:38.914366 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:49:38.914642 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.915153 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.916557 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.917040 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.917171 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:38.917234 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:38.917476 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 23:49:38.917544 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 23:49:38.918370 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:38.914302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:42115 (local address 127.6.152.3:33721)
0928 23:49:38.914443 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:38.914448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:38.914645 (+ 197us) server_negotiation.cc:407] Connection header received
0928 23:49:38.914760 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:38.914763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:38.914815 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:38.914897 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:49:38.915148 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.915764 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.917186 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.917794 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.917907 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:38.917991 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:38.918183 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 23:49:38.918257 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 23:49:38.918263 7467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d403ee7e344e1591aeb1312bc9904f" candidate_uuid: "8bb32ac1678f48d0bacca044aac7d05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2053e34c75ff4bc089e392c703f9f2f6" is_pre_election: true
I20260928 23:49:38.918629 7467 raft_consensus.cc:2454] T 57d403ee7e344e1591aeb1312bc9904f P 2053e34c75ff4bc089e392c703f9f2f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bb32ac1678f48d0bacca044aac7d05f in term 0.
I20260928 23:49:38.918738 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:38.914148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:33721 (local address 127.6.152.2:42115)
0928 23:49:38.914560 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:38.914570 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:38.914679 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:38.914906 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:38.914909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:38.914923 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:49:38.915046 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:38.915052 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.915900 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.915906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:38.917075 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:38.917084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.917930 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.917933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:38.918037 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:38.918060 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:49:38.918503 (+ 443us) client_negotiation.cc:240] Negotiation successful
0928 23:49:38.918587 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":138,"threads_started":1}
I20260928 23:49:38.919122 7535 leader_election.cc:302] T 57d403ee7e344e1591aeb1312bc9904f P 8bb32ac1678f48d0bacca044aac7d05f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2053e34c75ff4bc089e392c703f9f2f6, 8bb32ac1678f48d0bacca044aac7d05f; no voters:
I20260928 23:49:38.919420 7806 raft_consensus.cc:2781] T 57d403ee7e344e1591aeb1312bc9904f P 8bb32ac1678f48d0bacca044aac7d05f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:49:38.919418 7735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d403ee7e344e1591aeb1312bc9904f" candidate_uuid: "8bb32ac1678f48d0bacca044aac7d05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a7fe63093d410a9ba1a18b57c49f9d" is_pre_election: true
I20260928 23:49:38.919518 7806 raft_consensus.cc:482] T 57d403ee7e344e1591aeb1312bc9904f P 8bb32ac1678f48d0bacca044aac7d05f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:49:38.919576 7806 raft_consensus.cc:3037] T 57d403ee7e344e1591aeb1312bc9904f P 8bb32ac1678f48d0bacca044aac7d05f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:38.919679 7735 raft_consensus.cc:2454] T 57d403ee7e344e1591aeb1312bc9904f P 22a7fe63093d410a9ba1a18b57c49f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bb32ac1678f48d0bacca044aac7d05f in term 0.
I20260928 23:49:38.920732 7806 raft_consensus.cc:504] T 57d403ee7e344e1591aeb1312bc9904f P 8bb32ac1678f48d0bacca044aac7d05f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2053e34c75ff4bc089e392c703f9f2f6" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34617 } } peers { permanent_uuid: "22a7fe63093d410a9ba1a18b57c49f9d" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33721 } } peers { permanent_uuid: "8bb32ac1678f48d0bacca044aac7d05f" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 35707 } }
I20260928 23:49:38.921114 7806 leader_election.cc:288] T 57d403ee7e344e1591aeb1312bc9904f P 8bb32ac1678f48d0bacca044aac7d05f [CANDIDATE]: Term 1 election: Requested vote from peers 2053e34c75ff4bc089e392c703f9f2f6 (127.6.152.1:34617), 22a7fe63093d410a9ba1a18b57c49f9d (127.6.152.3:33721)
I20260928 23:49:38.921454 7735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d403ee7e344e1591aeb1312bc9904f" candidate_uuid: "8bb32ac1678f48d0bacca044aac7d05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a7fe63093d410a9ba1a18b57c49f9d"
I20260928 23:49:38.921550 7735 raft_consensus.cc:3037] T 57d403ee7e344e1591aeb1312bc9904f P 22a7fe63093d410a9ba1a18b57c49f9d [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:38.922513 7735 raft_consensus.cc:2454] T 57d403ee7e344e1591aeb1312bc9904f P 22a7fe63093d410a9ba1a18b57c49f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bb32ac1678f48d0bacca044aac7d05f in term 1.
I20260928 23:49:38.922736 7467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57d403ee7e344e1591aeb1312bc9904f" candidate_uuid: "8bb32ac1678f48d0bacca044aac7d05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2053e34c75ff4bc089e392c703f9f2f6"
I20260928 23:49:38.922797 7535 leader_election.cc:302] T 57d403ee7e344e1591aeb1312bc9904f P 8bb32ac1678f48d0bacca044aac7d05f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22a7fe63093d410a9ba1a18b57c49f9d, 8bb32ac1678f48d0bacca044aac7d05f; no voters:
I20260928 23:49:38.922895 7467 raft_consensus.cc:3037] T 57d403ee7e344e1591aeb1312bc9904f P 2053e34c75ff4bc089e392c703f9f2f6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:38.923013 7806 raft_consensus.cc:2781] T 57d403ee7e344e1591aeb1312bc9904f P 8bb32ac1678f48d0bacca044aac7d05f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:49:38.923249 7806 raft_consensus.cc:686] T 57d403ee7e344e1591aeb1312bc9904f P 8bb32ac1678f48d0bacca044aac7d05f [term 1 LEADER]: Becoming Leader. State: Replica: 8bb32ac1678f48d0bacca044aac7d05f, State: Running, Role: LEADER
I20260928 23:49:38.923830 7806 consensus_queue.cc:237] T 57d403ee7e344e1591aeb1312bc9904f P 8bb32ac1678f48d0bacca044aac7d05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2053e34c75ff4bc089e392c703f9f2f6" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34617 } } peers { permanent_uuid: "22a7fe63093d410a9ba1a18b57c49f9d" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33721 } } peers { permanent_uuid: "8bb32ac1678f48d0bacca044aac7d05f" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 35707 } }
I20260928 23:49:38.924129 7467 raft_consensus.cc:2454] T 57d403ee7e344e1591aeb1312bc9904f P 2053e34c75ff4bc089e392c703f9f2f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bb32ac1678f48d0bacca044aac7d05f in term 1.
I20260928 23:49:38.926968 7322 catalog_manager.cc:5495] T 57d403ee7e344e1591aeb1312bc9904f P 8bb32ac1678f48d0bacca044aac7d05f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bb32ac1678f48d0bacca044aac7d05f (127.6.152.2). New cstate: current_term: 1 leader_uuid: "8bb32ac1678f48d0bacca044aac7d05f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2053e34c75ff4bc089e392c703f9f2f6" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22a7fe63093d410a9ba1a18b57c49f9d" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bb32ac1678f48d0bacca044aac7d05f" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 35707 } health_report { overall_health: HEALTHY } } }
Sep 28 23:49:38 dist-test-slave-p5bw krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.1@KRBTEST.COM
I20260928 23:49:38.941566 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:38.932883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50972 (local address 127.6.152.1:34617)
0928 23:49:38.933014 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:38.933019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:38.933212 (+ 193us) server_negotiation.cc:407] Connection header received
0928 23:49:38.933362 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:38.933366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:38.933415 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:38.933500 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:38.934157 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.934648 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.935467 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.935683 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.935788 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:38.938345 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:38.938362 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:38.938375 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:38.938397 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:38.939772 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:38.940224 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:38.940236 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:38.940238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:38.940319 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:38.940600 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:38.940603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:38.940605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:38.940914 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:38.941115 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:38.941282 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 23:49:38.941400 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 23:49:38 dist-test-slave-p5bw krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639378, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 23:49:38 dist-test-slave-p5bw krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639378, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.152.1@KRBTEST.COM
I20260928 23:49:38.967216 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:38.959366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50984 (local address 127.6.152.1:34617)
0928 23:49:38.959553 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:38.959557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:38.959612 (+ 55us) server_negotiation.cc:407] Connection header received
0928 23:49:38.959722 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:38.959725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:38.959764 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:38.959843 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:38.960456 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.961051 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.961849 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.962072 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.962161 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:38.964547 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:38.964550 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:38.964552 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:38.964574 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:38.965853 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:38.966231 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:38.966234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:38.966235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:38.966304 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:38.966522 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:38.966524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:38.966525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:38.966670 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:38.966765 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:38.967007 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 23:49:38.967098 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Sep 28 23:49:38 dist-test-slave-p5bw krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 23:49:38 dist-test-slave-p5bw krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.1@KRBTEST.COM
I20260928 23:49:38.991739 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:38.983676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50992 (local address 127.6.152.1:34617)
0928 23:49:38.983845 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:38.983849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:38.983985 (+ 136us) server_negotiation.cc:407] Connection header received
0928 23:49:38.984139 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:38.984142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:38.984192 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:38.984276 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:38.984968 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.985471 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.986302 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:38.986525 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:38.986633 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:38.989053 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:38.989056 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:38.989058 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:38.989081 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:38.990377 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:38.990754 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:38.990757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:38.990759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:38.990844 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:38.991080 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:38.991082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:38.991083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:38.991243 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:38.991342 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:38.991532 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 23:49:38.991624 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 23:49:38.993192 7427 tablet_service.cc:2908] Rejecting scan request for tablet 57d403ee7e344e1591aeb1312bc9904f: Uninitialized: clean time has not yet been initialized
I20260928 23:49:38.995149 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 7384
I20260928 23:49:39.000787 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 7518
I20260928 23:49:39.006356 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 7652
I20260928 23:49:39.011343 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 7290
2026-09-28T23:49:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2249 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:49:39 dist-test-slave-p5bw krb5kdc[7822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:49:39 dist-test-slave-p5bw krb5kdc[7822](info): set up 2 sockets
Sep 28 23:49:39 dist-test-slave-p5bw krb5kdc[7822](info): commencing operation
krb5kdc: starting...
W20260928 23:49:40.041292 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:49:40 dist-test-slave-p5bw krb5kdc[7822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:49:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:49:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:49:40.159921 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:34703
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:42413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:34703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:40.265853 7835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:40.266140 7835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:40.266211 7835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:40.270134 7835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:49:40.270216 7835 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 23:49:40.270248 7835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:40.270287 7835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:40.270323 7835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:49:40.270358 7835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:49:40.275452 7835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:42413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:34703
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:34703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:40.276472 7835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:40.277485 7835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:40.285768 7843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:40.285797 7841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:40.285799 7840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:40.286059 7835 server_base.cc:1034] running on GCE node
I20260928 23:49:40.286728 7835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:40.287860 7835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:40.289055 7835 hybrid_clock.cc:648] HybridClock initialized: now 1790639380289023 us; error 46 us; skew 500 ppm
Sep 28 23:49:40 dist-test-slave-p5bw krb5kdc[7822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639380, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:40.298496 7835 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:49:40.299459 7835 webserver.cc:460] Webserver started at http://127.6.152.62:46421/ using document root <none> and password file <none>
I20260928 23:49:40.300034 7835 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:40.300125 7835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:40.300354 7835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:40.302425 7835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "a1d002a27b574ffd969ddfcb9fd58ca4"
format_stamp: "Formatted at 2026-09-28 23:49:40 on dist-test-slave-p5bw"
server_key: "5adf12415f0957ceb06a69675c27ed48"
server_key_iv: "b32bda5fe161198b8461cf176dd3eb5d"
server_key_version: "encryptionkey@0"
I20260928 23:49:40.302999 7835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "a1d002a27b574ffd969ddfcb9fd58ca4"
format_stamp: "Formatted at 2026-09-28 23:49:40 on dist-test-slave-p5bw"
server_key: "5adf12415f0957ceb06a69675c27ed48"
server_key_iv: "b32bda5fe161198b8461cf176dd3eb5d"
server_key_version: "encryptionkey@0"
I20260928 23:49:40.306710 7835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 23:49:40.309139 7850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:40.310453 7835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:49:40.310595 7835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "a1d002a27b574ffd969ddfcb9fd58ca4"
format_stamp: "Formatted at 2026-09-28 23:49:40 on dist-test-slave-p5bw"
server_key: "5adf12415f0957ceb06a69675c27ed48"
server_key_iv: "b32bda5fe161198b8461cf176dd3eb5d"
server_key_version: "encryptionkey@0"
I20260928 23:49:40.310711 7835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:40.330255 7835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:40.333139 7835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:40.333350 7835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:40.341444 7835 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:34703
I20260928 23:49:40.341451 7912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:34703 every 8 connection(s)
I20260928 23:49:40.342391 7835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:49:40.345153 7913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:40.345826 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 7835
I20260928 23:49:40.345984 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:49:40.346248 6752 external_mini_cluster.cc:1428] Setting key 70f5386b75237de49a40434d760dc762
I20260928 23:49:40.351738 7913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4: Bootstrap starting.
I20260928 23:49:40.354218 7913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:40.354921 7913 log.cc:824] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:49:40 dist-test-slave-p5bw krb5kdc[7822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:40.357069 7913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4: No bootstrap required, opened a new log
I20260928 23:49:40.361284 7913 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d002a27b574ffd969ddfcb9fd58ca4" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 34703 } }
I20260928 23:49:40.361545 7913 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:40.361601 7913 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1d002a27b574ffd969ddfcb9fd58ca4, State: Initialized, Role: FOLLOWER
I20260928 23:49:40.362056 7913 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d002a27b574ffd969ddfcb9fd58ca4" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 34703 } }
I20260928 23:49:40.362205 7913 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:49:40.362268 7913 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:49:40.362350 7913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:40.362394 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:40.347634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40252 (local address 127.6.152.62:34703)
0928 23:49:40.348063 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:40.348081 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:40.348126 (+ 45us) server_negotiation.cc:407] Connection header received
0928 23:49:40.348964 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:40.348987 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:40.349388 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:40.349752 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:40.350453 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:40.351889 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:40.352731 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:40.353425 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:40.353591 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:40.356385 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:40.356391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:40.356407 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:40.356436 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:40.359375 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:40.359789 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:40.359795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:40.359800 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:40.359934 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:40.360187 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:40.360190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:40.360191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:40.360687 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:40.360895 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:40.361156 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 23:49:40.361469 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":111,"threads_started":1}
I20260928 23:49:40.363227 7913 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d002a27b574ffd969ddfcb9fd58ca4" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 34703 } }
I20260928 23:49:40.363565 7913 leader_election.cc:302] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1d002a27b574ffd969ddfcb9fd58ca4; no voters:
I20260928 23:49:40.363878 7913 leader_election.cc:288] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:49:40.363951 7918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:49:40.364161 7918 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4 [term 1 LEADER]: Becoming Leader. State: Replica: a1d002a27b574ffd969ddfcb9fd58ca4, State: Running, Role: LEADER
I20260928 23:49:40.364449 7918 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d002a27b574ffd969ddfcb9fd58ca4" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 34703 } }
I20260928 23:49:40.365146 7913 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:49:40.365593 7920 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1d002a27b574ffd969ddfcb9fd58ca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d002a27b574ffd969ddfcb9fd58ca4" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 34703 } } }
I20260928 23:49:40.365716 7920 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4 [sys.catalog]: This master's current role is: LEADER
I20260928 23:49:40.366073 7919 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1d002a27b574ffd969ddfcb9fd58ca4. Latest consensus state: current_term: 1 leader_uuid: "a1d002a27b574ffd969ddfcb9fd58ca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d002a27b574ffd969ddfcb9fd58ca4" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 34703 } } }
I20260928 23:49:40.366189 7919 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4 [sys.catalog]: This master's current role is: LEADER
W20260928 23:49:40.370061 7934 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:49:40.370136 7934 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:49:40.370208 7922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:49:40.371001 7922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:49:40.376912 7922 catalog_manager.cc:1347] Generated new cluster ID: cfffdf8eb7d94a13a738058854232988
I20260928 23:49:40.376979 7922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:49:40.393097 7922 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:49:40.394200 7922 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:49:40.410409 7922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4: Generated new TSK 0
I20260928 23:49:40.411168 7922 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
I20260928 23:49:40.445713 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:34703
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:42413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:40.553592 7939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:40.553829 7939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:40.553874 7939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:40.557842 7939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:40.557893 7939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:40.557956 7939 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:49:40.558019 7939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:49:40.563120 7939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:42413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.152.62:34703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:40.564090 7939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:40.565101 7939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:40.574365 7947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:40.574448 7944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:40.574412 7945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:40.574570 7939 server_base.cc:1034] running on GCE node
I20260928 23:49:40.575033 7939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:40.575685 7939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:40.576879 7939 hybrid_clock.cc:648] HybridClock initialized: now 1790639380576858 us; error 31 us; skew 500 ppm
Sep 28 23:49:40 dist-test-slave-p5bw krb5kdc[7822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639380, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:40.579392 7939 init.cc:377] Logged in from keytab as kudu/127.6.152.1@KRBTEST.COM (short username kudu)
I20260928 23:49:40.580178 7939 webserver.cc:460] Webserver started at http://127.6.152.1:39821/ using document root <none> and password file <none>
I20260928 23:49:40.580726 7939 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:40.580813 7939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:40.581043 7939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:40.582957 7939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "1e4d4ae1ad144d128642390b850e6b89"
format_stamp: "Formatted at 2026-09-28 23:49:40 on dist-test-slave-p5bw"
server_key: "5377f657e2860880b924431fdcf2b719"
server_key_iv: "2b2e40b96c3cf37bfa7a9ed83f3e0758"
server_key_version: "encryptionkey@0"
I20260928 23:49:40.583492 7939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "1e4d4ae1ad144d128642390b850e6b89"
format_stamp: "Formatted at 2026-09-28 23:49:40 on dist-test-slave-p5bw"
server_key: "5377f657e2860880b924431fdcf2b719"
server_key_iv: "2b2e40b96c3cf37bfa7a9ed83f3e0758"
server_key_version: "encryptionkey@0"
I20260928 23:49:40.586935 7939 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 23:49:40.589190 7954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:40.590327 7939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:49:40.590456 7939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "1e4d4ae1ad144d128642390b850e6b89"
format_stamp: "Formatted at 2026-09-28 23:49:40 on dist-test-slave-p5bw"
server_key: "5377f657e2860880b924431fdcf2b719"
server_key_iv: "2b2e40b96c3cf37bfa7a9ed83f3e0758"
server_key_version: "encryptionkey@0"
I20260928 23:49:40.590555 7939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:40.610666 7939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:40.613653 7939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:40.613839 7939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:40.615340 7939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:40.615409 7939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:40.615473 7939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:40.615502 7939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:49:40 dist-test-slave-p5bw krb5kdc[7822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639380, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:40.626184 7939 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:42659
I20260928 23:49:40.626241 8069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:42659 every 8 connection(s)
I20260928 23:49:40.627091 7939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:49:40.628981 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:40.616341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:58881 (local address 127.6.152.62:34703)
0928 23:49:40.616495 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:40.616502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:40.617403 (+ 901us) server_negotiation.cc:407] Connection header received
0928 23:49:40.618707 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:40.618712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:40.618775 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:40.618873 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:40.620499 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:40.621275 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:40.622422 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:40.622673 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:40.622805 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:40.625387 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:40.625391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:40.625394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:40.625421 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:40.626997 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:40.627530 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:40.627534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:40.627536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:40.627618 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:40.627996 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:40.627999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:40.628001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:40.628203 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:40.628280 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:40.628701 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 23:49:40.628798 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:49:40.629760 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:40.616696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:34703 (local address 127.6.152.1:58881)
0928 23:49:40.617226 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:40.617282 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:40.618451 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:40.619091 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:40.619103 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:40.619591 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:40.620307 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:40.620330 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:40.621416 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:40.621419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:40.622292 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:40.622302 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:40.622851 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:40.622855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:40.623024 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:40.623608 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:40.623623 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:40.625220 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:40.627119 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:40.627125 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:40.627139 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:40.627426 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:40.627716 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:40.627719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:40.627721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:40.627895 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:40.628295 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:40.628300 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:40.628506 (+ 206us) client_negotiation.cc:769] Sending connection context
0928 23:49:40.628654 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 23:49:40.628971 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":162,"threads_started":1}
I20260928 23:49:40.631006 8070 heartbeater.cc:347] Connected to a master server at 127.6.152.62:34703
I20260928 23:49:40.631162 8070 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:40.631556 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 7939
I20260928 23:49:40.631671 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
I20260928 23:49:40.631733 8070 heartbeater.cc:511] Master 127.6.152.62:34703 requested a full tablet report, sending...
I20260928 23:49:40.631963 6752 external_mini_cluster.cc:1428] Setting key 795ddc7dc8ac22aa930e6935f6d89d33
I20260928 23:49:40.633445 7866 ts_manager.cc:194] Registered new tserver with Master: 1e4d4ae1ad144d128642390b850e6b89 (127.6.152.1:42659)
I20260928 23:49:40.634537 7866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.1@KRBTEST.COM'} at 127.6.152.1:58881
I20260928 23:49:40.641506 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:40.633223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:34007 (local address 127.6.152.62:34703)
0928 23:49:40.633331 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:40.633335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:40.633595 (+ 260us) server_negotiation.cc:407] Connection header received
0928 23:49:40.633808 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:40.633810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:40.633851 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:40.633904 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:40.634586 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:40.635404 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:40.636733 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:40.637012 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:40.637094 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:40.638383 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:40.638386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:40.638387 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:40.638407 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:40.639829 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:40.640324 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:40.640327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:40.640328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:40.640405 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:40.640696 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:40.640698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:40.640699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:40.640911 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:40.640987 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:40.641297 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 23:49:40.641396 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 23:49:40.641870 8076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:40.633189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:34703 (local address 127.6.152.1:34007)
0928 23:49:40.633479 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:40.633493 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:40.633693 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:40.634067 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:40.634072 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:40.634302 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:40.634450 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:40.634461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:40.635615 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:40.635620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:40.636609 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:40.636623 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:40.637126 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:40.637133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:40.637265 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:40.637737 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:40.637757 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:40.638248 (+ 491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:40.639944 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:40.639949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:40.639951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:40.640222 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:40.640497 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:40.640501 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:40.640503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:40.640601 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:40.641012 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:40.641018 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:40.641127 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 23:49:40.641241 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 23:49:40.641395 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":3,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.6.152.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
I20260928 23:49:40.657924 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:34703
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:42413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:40.764909 8079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:40.765185 8079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:40.765228 8079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:40.769387 8079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:40.769439 8079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:40.769477 8079 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:49:40.769512 8079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:49:40.774469 8079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:42413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.152.62:34703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:40.775382 8079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:40.776286 8079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:40.785586 8085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:40.785648 8079 server_base.cc:1034] running on GCE node
W20260928 23:49:40.785574 8084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:40.785809 8087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:40.786239 8079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:40.786834 8079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:40.787998 8079 hybrid_clock.cc:648] HybridClock initialized: now 1790639380787978 us; error 33 us; skew 500 ppm
Sep 28 23:49:40 dist-test-slave-p5bw krb5kdc[7822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639380, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:40.790683 8079 init.cc:377] Logged in from keytab as kudu/127.6.152.2@KRBTEST.COM (short username kudu)
I20260928 23:49:40.791512 8079 webserver.cc:460] Webserver started at http://127.6.152.2:44495/ using document root <none> and password file <none>
I20260928 23:49:40.792055 8079 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:40.792150 8079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:40.792385 8079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:40.794708 8079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "3b234a727a7a4bafa8d69622cf3460fb"
format_stamp: "Formatted at 2026-09-28 23:49:40 on dist-test-slave-p5bw"
server_key: "e445a8ec49916a622845145b436c1132"
server_key_iv: "41b0c12c427bc90e458daca6f362d8af"
server_key_version: "encryptionkey@0"
I20260928 23:49:40.795265 8079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "3b234a727a7a4bafa8d69622cf3460fb"
format_stamp: "Formatted at 2026-09-28 23:49:40 on dist-test-slave-p5bw"
server_key: "e445a8ec49916a622845145b436c1132"
server_key_iv: "41b0c12c427bc90e458daca6f362d8af"
server_key_version: "encryptionkey@0"
I20260928 23:49:40.798930 8079 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 23:49:40.801399 8094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:40.802524 8079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:49:40.802663 8079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "3b234a727a7a4bafa8d69622cf3460fb"
format_stamp: "Formatted at 2026-09-28 23:49:40 on dist-test-slave-p5bw"
server_key: "e445a8ec49916a622845145b436c1132"
server_key_iv: "41b0c12c427bc90e458daca6f362d8af"
server_key_version: "encryptionkey@0"
I20260928 23:49:40.802798 8079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:40.826154 8079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:40.829449 8079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:40.829609 8079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:40.830937 8079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:40.830982 8079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:40.831049 8079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:40.831089 8079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:49:40 dist-test-slave-p5bw krb5kdc[7822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639380, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:40.842157 8079 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:46359
I20260928 23:49:40.842243 8209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:46359 every 8 connection(s)
I20260928 23:49:40.843360 8079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:49:40.845652 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:40.831838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:48143 (local address 127.6.152.62:34703)
0928 23:49:40.831970 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:40.831974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:40.833062 (+ 1088us) server_negotiation.cc:407] Connection header received
0928 23:49:40.834189 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:40.834193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:40.834255 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:40.834358 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:40.835976 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:40.836872 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:40.838779 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:40.839104 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:40.839196 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:40.841932 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:40.841936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:40.841940 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:40.842031 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:40.843601 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:40.844195 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:40.844200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:40.844201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:40.844299 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:40.844647 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:40.844649 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:40.844650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:40.844884 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:40.844975 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:40.845422 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 23:49:40.845530 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:49:40.846413 8103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:40.832386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:34703 (local address 127.6.152.2:48143)
0928 23:49:40.832911 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:40.832952 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:40.833924 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:40.834560 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:40.834570 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:40.835087 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:40.835798 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:40.835823 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:40.837012 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:40.837017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:40.838616 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:40.838632 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:40.839275 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:40.839281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:40.839511 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:40.840180 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:40.840206 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:40.841684 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:40.843731 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:40.843739 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:40.843754 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:40.844088 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:40.844413 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:40.844416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:40.844417 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:40.844552 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:40.844983 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:40.844988 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:40.845207 (+ 219us) client_negotiation.cc:769] Sending connection context
0928 23:49:40.845360 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 23:49:40.845599 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":134,"threads_started":1}
I20260928 23:49:40.847724 8210 heartbeater.cc:347] Connected to a master server at 127.6.152.62:34703
I20260928 23:49:40.847900 8210 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:40.848356 8210 heartbeater.cc:511] Master 127.6.152.62:34703 requested a full tablet report, sending...
I20260928 23:49:40.849429 7867 ts_manager.cc:194] Registered new tserver with Master: 3b234a727a7a4bafa8d69622cf3460fb (127.6.152.2:46359)
I20260928 23:49:40.850203 7867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.2@KRBTEST.COM'} at 127.6.152.2:48143
I20260928 23:49:40.853614 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 8079
I20260928 23:49:40.853739 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:49:40.854054 6752 external_mini_cluster.cc:1428] Setting key ce6f82c663bb4048026f3e7169463b18
I20260928 23:49:40.856729 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:40.849767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:36179 (local address 127.6.152.62:34703)
0928 23:49:40.849891 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:40.849895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:40.850072 (+ 177us) server_negotiation.cc:407] Connection header received
0928 23:49:40.850167 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:40.850169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:40.850203 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:40.850264 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:40.850714 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:40.851222 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:40.852259 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:40.852456 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:40.852532 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:40.853534 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:40.853539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:40.853541 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:40.853570 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:40.855057 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:40.855530 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:40.855533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:40.855537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:40.855611 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:40.855888 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:40.855894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:40.855896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:40.856093 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:40.856196 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:40.856462 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 23:49:40.856582 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:49:40.856736 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:40.849712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:34703 (local address 127.6.152.2:36179)
0928 23:49:40.849990 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:40.850005 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:40.850091 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:40.850307 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:40.850310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:40.850519 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:40.850610 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:40.850616 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:40.851353 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:40.851356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:40.852146 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:40.852155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:40.852543 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:40.852545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:40.852610 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:40.853078 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:40.853088 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:40.853390 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:40.855166 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:40.855172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:40.855174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:40.855413 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:40.855703 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:40.855709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:40.855711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:40.855807 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:40.856197 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:40.856201 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:40.856294 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 23:49:40.856398 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 23:49:40.856544 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.6.152.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
I20260928 23:49:40.879769 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:34703
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:42413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:40.988114 8219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:40.988350 8219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:40.988394 8219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:40.992223 8219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:40.992267 8219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:40.992303 8219 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:49:40.992334 8219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:49:40.997468 8219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:42413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.152.62:34703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:40.998394 8219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:40.999290 8219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:41.008648 8224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:41.008630 8225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:41.008630 8227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:41.009045 8219 server_base.cc:1034] running on GCE node
I20260928 23:49:41.009459 8219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:41.010110 8219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:41.011291 8219 hybrid_clock.cc:648] HybridClock initialized: now 1790639381011272 us; error 41 us; skew 500 ppm
Sep 28 23:49:41 dist-test-slave-p5bw krb5kdc[7822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639381, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:41.013988 8219 init.cc:377] Logged in from keytab as kudu/127.6.152.3@KRBTEST.COM (short username kudu)
I20260928 23:49:41.014843 8219 webserver.cc:460] Webserver started at http://127.6.152.3:33103/ using document root <none> and password file <none>
I20260928 23:49:41.015384 8219 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:41.015442 8219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:41.015597 8219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:41.017622 8219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "93ea9cdce01f4b73bd502aa818473a43"
format_stamp: "Formatted at 2026-09-28 23:49:41 on dist-test-slave-p5bw"
server_key: "43a2b086cbed04c06e4374adecc34926"
server_key_iv: "1d9f481bb22deae35794b8b0d225ce5b"
server_key_version: "encryptionkey@0"
I20260928 23:49:41.018133 8219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "93ea9cdce01f4b73bd502aa818473a43"
format_stamp: "Formatted at 2026-09-28 23:49:41 on dist-test-slave-p5bw"
server_key: "43a2b086cbed04c06e4374adecc34926"
server_key_iv: "1d9f481bb22deae35794b8b0d225ce5b"
server_key_version: "encryptionkey@0"
I20260928 23:49:41.021672 8219 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:49:41.023989 8234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:41.025244 8219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 23:49:41.025334 8219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "93ea9cdce01f4b73bd502aa818473a43"
format_stamp: "Formatted at 2026-09-28 23:49:41 on dist-test-slave-p5bw"
server_key: "43a2b086cbed04c06e4374adecc34926"
server_key_iv: "1d9f481bb22deae35794b8b0d225ce5b"
server_key_version: "encryptionkey@0"
I20260928 23:49:41.025416 8219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:41.052796 8219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:41.055527 8219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:41.055673 8219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:41.057061 8219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:41.057113 8219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:41.057181 8219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:41.057216 8219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:49:41 dist-test-slave-p5bw krb5kdc[7822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639381, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:41.068073 8219 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:40847
I20260928 23:49:41.068130 8349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:40847 every 8 connection(s)
I20260928 23:49:41.069128 8219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:49:41.071823 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:41.057885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:51189 (local address 127.6.152.62:34703)
0928 23:49:41.058059 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:41.058065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:41.058871 (+ 806us) server_negotiation.cc:407] Connection header received
0928 23:49:41.059892 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:41.059897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:41.059959 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:41.060034 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:41.061679 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.062585 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.064012 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.064412 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.064524 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:41.068515 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:41.068518 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:41.068520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:41.068544 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:41.070169 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:41.070587 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:41.070589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:41.070590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:41.070659 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:41.070925 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:41.070927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:41.070928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:41.071104 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:41.071180 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:41.071612 (+ 432us) server_negotiation.cc:299] Negotiation successful
0928 23:49:41.071706 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 23:49:41.072599 8243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:41.058260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:34703 (local address 127.6.152.3:51189)
0928 23:49:41.058701 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:41.058750 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:41.059645 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:41.060513 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:41.060520 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:41.061003 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:41.061523 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:41.061537 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.062700 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.062703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:41.063775 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:41.063790 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.064710 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.064717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:41.065084 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:41.066457 (+ 1373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:41.066487 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:41.068349 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:41.070247 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:41.070250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:41.070260 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:41.070494 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:41.070717 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:41.070718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:41.070720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:41.070831 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:41.071194 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:41.071199 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:41.071406 (+ 207us) client_negotiation.cc:769] Sending connection context
0928 23:49:41.071567 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 23:49:41.071799 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":137,"threads_started":1}
I20260928 23:49:41.073760 8350 heartbeater.cc:347] Connected to a master server at 127.6.152.62:34703
I20260928 23:49:41.073892 8350 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:41.074429 8350 heartbeater.cc:511] Master 127.6.152.62:34703 requested a full tablet report, sending...
I20260928 23:49:41.075484 7866 ts_manager.cc:194] Registered new tserver with Master: 93ea9cdce01f4b73bd502aa818473a43 (127.6.152.3:40847)
I20260928 23:49:41.075731 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 8219
I20260928 23:49:41.075851 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:49:41.076165 6752 external_mini_cluster.cc:1428] Setting key 69889aace1c72eea44695e87c6e9630c
I20260928 23:49:41.076263 7866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.3@KRBTEST.COM'} at 127.6.152.3:51189
I20260928 23:49:41.078294 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:49:41.084250 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:41.075980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:34703 (local address 127.6.152.3:44565)
0928 23:49:41.076755 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:41.076770 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:41.076911 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:41.077252 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:41.077255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:41.077485 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:41.077624 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:41.077631 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.078508 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.078511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:41.079399 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:41.079413 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.079852 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.079858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:41.079995 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:41.080537 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:41.080549 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:41.080905 (+ 356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:41.082686 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:41.082692 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:41.082697 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:41.082989 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:41.083280 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:41.083286 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:41.083290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:41.083385 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:41.083839 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:41.083841 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:41.083915 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 23:49:41.084006 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 23:49:41.084109 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":3,"thread_start_us":98,"threads_started":1}
I20260928 23:49:41.084476 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:41.076082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:44565 (local address 127.6.152.62:34703)
0928 23:49:41.076190 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:41.076195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:41.076848 (+ 653us) server_negotiation.cc:407] Connection header received
0928 23:49:41.077022 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:41.077027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:41.077070 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:41.077120 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:41.077732 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.078368 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.079538 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.079742 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.079818 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:41.081051 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:41.081055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:41.081057 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:41.081084 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:41.082556 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:41.083090 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:41.083094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:41.083095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:41.083175 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:41.083499 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:41.083504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:41.083505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:41.083736 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:41.083843 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:41.084104 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 23:49:41.084236 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 23:49:41 dist-test-slave-p5bw krb5kdc[7822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639381, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 23:49:41 dist-test-slave-p5bw krb5kdc[7822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639381, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:41.104476 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:41.096252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40258 (local address 127.6.152.62:34703)
0928 23:49:41.096392 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:41.096395 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:41.096407 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:49:41.096440 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:41.096442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:41.096478 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:41.096577 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:41.097382 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.098178 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.099032 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.099249 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.099359 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:41.101664 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:41.101667 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:41.101669 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:41.101701 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:41.102854 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:41.103274 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:41.103277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:41.103279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:41.103355 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:41.103638 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:41.103641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:41.103643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:41.103805 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:41.103961 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:41.104200 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 23:49:41.104314 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 23:49:41.107234 7866 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:49:41.110172 7866 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:49:41.124943 8369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:41.118647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36844 (local address 127.6.152.1:42659)
0928 23:49:41.120481 (+ 1834us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:41.120496 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:41.120517 (+ 21us) server_negotiation.cc:407] Connection header received
0928 23:49:41.120583 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:41.120588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:41.120748 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:41.120924 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:49:41.121130 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.122149 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.123376 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.124126 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.124236 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:41.124405 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:41.124672 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 23:49:41.124768 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1731,"thread_start_us":77,"threads_started":1}
I20260928 23:49:41.124986 8368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:41.118492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:42659 (local address 127.0.0.1:36844)
0928 23:49:41.119273 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:41.119320 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:41.119458 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:41.120903 (+ 1445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:41.120906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:41.120924 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:49:41.121011 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:41.121017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.122265 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.122269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:41.123211 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:41.123221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.124249 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.124253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:41.124430 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:41.124513 (+ 83us) client_negotiation.cc:769] Sending connection context
0928 23:49:41.124621 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 23:49:41.124801 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":107,"spinlock_wait_cycles":6656,"thread_start_us":98,"threads_started":1}
I20260928 23:49:41.124986 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:41.118823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:40847 (local address 127.0.0.1:59116)
0928 23:49:41.119677 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:41.119687 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:41.119787 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:41.120358 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:41.120361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:41.120378 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:49:41.120494 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:41.120500 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.121604 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.121610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:41.123193 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:41.123204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.124249 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.124253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:41.124435 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:41.124509 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 23:49:41.124621 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 23:49:41.124801 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":185,"thread_start_us":87,"threads_started":1}
I20260928 23:49:41.125432 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:41.119266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59116 (local address 127.6.152.3:40847)
0928 23:49:41.120011 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:41.120020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:41.120037 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:49:41.120094 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:41.120097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:41.120223 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:41.120378 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:49:41.120613 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.121491 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.123322 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.124125 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.124236 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:41.124398 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:41.125181 (+ 783us) server_negotiation.cc:299] Negotiation successful
0928 23:49:41.125273 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":102,"threads_started":1}
I20260928 23:49:41.126506 8367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:41.118544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:46359 (local address 127.0.0.1:57220)
0928 23:49:41.119281 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:41.119321 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:41.119459 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:41.120179 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:41.120185 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:41.120206 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:49:41.120337 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:41.120351 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.121793 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.121796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:41.123193 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:41.123204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.126103 (+ 2899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.126107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:41.126204 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:41.126228 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 23:49:41.126314 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 23:49:41.126382 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":63,"spinlock_wait_cycles":3200,"thread_start_us":95,"threads_started":1}
I20260928 23:49:41.126564 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:41.119062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57220 (local address 127.6.152.2:46359)
0928 23:49:41.119730 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:41.119744 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:41.119766 (+ 22us) server_negotiation.cc:407] Connection header received
0928 23:49:41.119838 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:41.119845 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:41.120031 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:41.120191 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:49:41.120469 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.121682 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.124459 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.125980 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.126084 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:41.126230 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:41.126338 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 23:49:41.126422 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":90,"threads_started":1}
I20260928 23:49:41.128134 8004 tablet_service.cc:1467] Processing CreateTablet for tablet 385b424413e64e6597e247a8b47cc694 (DEFAULT_TABLE table=test-table [id=ce668213c76748109be68f435d8efe39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:41.128095 8284 tablet_service.cc:1467] Processing CreateTablet for tablet 385b424413e64e6597e247a8b47cc694 (DEFAULT_TABLE table=test-table [id=ce668213c76748109be68f435d8efe39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:41.129207 8004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385b424413e64e6597e247a8b47cc694. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:41.129207 8284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385b424413e64e6597e247a8b47cc694. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:41.129168 8144 tablet_service.cc:1467] Processing CreateTablet for tablet 385b424413e64e6597e247a8b47cc694 (DEFAULT_TABLE table=test-table [id=ce668213c76748109be68f435d8efe39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:41.130019 8144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385b424413e64e6597e247a8b47cc694. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:41.135995 8373 tablet_bootstrap.cc:492] T 385b424413e64e6597e247a8b47cc694 P 93ea9cdce01f4b73bd502aa818473a43: Bootstrap starting.
I20260928 23:49:41.137127 8374 tablet_bootstrap.cc:492] T 385b424413e64e6597e247a8b47cc694 P 1e4d4ae1ad144d128642390b850e6b89: Bootstrap starting.
I20260928 23:49:41.137650 8375 tablet_bootstrap.cc:492] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb: Bootstrap starting.
I20260928 23:49:41.138901 8373 tablet_bootstrap.cc:654] T 385b424413e64e6597e247a8b47cc694 P 93ea9cdce01f4b73bd502aa818473a43: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:41.139371 8375 tablet_bootstrap.cc:654] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:41.139596 8374 tablet_bootstrap.cc:654] T 385b424413e64e6597e247a8b47cc694 P 1e4d4ae1ad144d128642390b850e6b89: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:41.139755 8373 log.cc:824] T 385b424413e64e6597e247a8b47cc694 P 93ea9cdce01f4b73bd502aa818473a43: Log is configured to *not* fsync() on all Append() calls
I20260928 23:49:41.140125 8375 log.cc:824] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb: Log is configured to *not* fsync() on all Append() calls
I20260928 23:49:41.140365 8374 log.cc:824] T 385b424413e64e6597e247a8b47cc694 P 1e4d4ae1ad144d128642390b850e6b89: Log is configured to *not* fsync() on all Append() calls
I20260928 23:49:41.141842 8375 tablet_bootstrap.cc:492] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb: No bootstrap required, opened a new log
I20260928 23:49:41.141842 8373 tablet_bootstrap.cc:492] T 385b424413e64e6597e247a8b47cc694 P 93ea9cdce01f4b73bd502aa818473a43: No bootstrap required, opened a new log
I20260928 23:49:41.142057 8373 ts_tablet_manager.cc:1397] T 385b424413e64e6597e247a8b47cc694 P 93ea9cdce01f4b73bd502aa818473a43: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:49:41.142194 8375 ts_tablet_manager.cc:1397] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 23:49:41.142217 8374 tablet_bootstrap.cc:492] T 385b424413e64e6597e247a8b47cc694 P 1e4d4ae1ad144d128642390b850e6b89: No bootstrap required, opened a new log
I20260928 23:49:41.142405 8374 ts_tablet_manager.cc:1397] T 385b424413e64e6597e247a8b47cc694 P 1e4d4ae1ad144d128642390b850e6b89: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 23:49:41.145301 8374 raft_consensus.cc:348] T 385b424413e64e6597e247a8b47cc694 P 1e4d4ae1ad144d128642390b850e6b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.145524 8374 raft_consensus.cc:374] T 385b424413e64e6597e247a8b47cc694 P 1e4d4ae1ad144d128642390b850e6b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:41.145593 8374 raft_consensus.cc:729] T 385b424413e64e6597e247a8b47cc694 P 1e4d4ae1ad144d128642390b850e6b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4d4ae1ad144d128642390b850e6b89, State: Initialized, Role: FOLLOWER
I20260928 23:49:41.146019 8374 consensus_queue.cc:260] T 385b424413e64e6597e247a8b47cc694 P 1e4d4ae1ad144d128642390b850e6b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.145969 8373 raft_consensus.cc:348] T 385b424413e64e6597e247a8b47cc694 P 93ea9cdce01f4b73bd502aa818473a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.145969 8375 raft_consensus.cc:348] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.146205 8375 raft_consensus.cc:374] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:41.146232 8373 raft_consensus.cc:374] T 385b424413e64e6597e247a8b47cc694 P 93ea9cdce01f4b73bd502aa818473a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:41.146293 8375 raft_consensus.cc:729] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b234a727a7a4bafa8d69622cf3460fb, State: Initialized, Role: FOLLOWER
I20260928 23:49:41.146306 8373 raft_consensus.cc:729] T 385b424413e64e6597e247a8b47cc694 P 93ea9cdce01f4b73bd502aa818473a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93ea9cdce01f4b73bd502aa818473a43, State: Initialized, Role: FOLLOWER
I20260928 23:49:41.146726 8373 consensus_queue.cc:260] T 385b424413e64e6597e247a8b47cc694 P 93ea9cdce01f4b73bd502aa818473a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.146726 8375 consensus_queue.cc:260] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.146911 8374 ts_tablet_manager.cc:1428] T 385b424413e64e6597e247a8b47cc694 P 1e4d4ae1ad144d128642390b850e6b89: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260928 23:49:41.147287 8070 heartbeater.cc:503] Master 127.6.152.62:34703 was elected leader, sending a full tablet report...
I20260928 23:49:41.147616 8375 ts_tablet_manager.cc:1428] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 23:49:41.147616 8373 ts_tablet_manager.cc:1428] T 385b424413e64e6597e247a8b47cc694 P 93ea9cdce01f4b73bd502aa818473a43: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 23:49:41.147928 8210 heartbeater.cc:503] Master 127.6.152.62:34703 was elected leader, sending a full tablet report...
I20260928 23:49:41.148615 8350 heartbeater.cc:503] Master 127.6.152.62:34703 was elected leader, sending a full tablet report...
I20260928 23:49:41.277328 8380 raft_consensus.cc:482] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:49:41.277575 8380 raft_consensus.cc:504] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.278771 8380 leader_election.cc:288] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e4d4ae1ad144d128642390b850e6b89 (127.6.152.1:42659), 93ea9cdce01f4b73bd502aa818473a43 (127.6.152.3:40847)
I20260928 23:49:41.283737 8369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:41.279175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:52507 (local address 127.6.152.1:42659)
0928 23:49:41.279305 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:41.279310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:41.279726 (+ 416us) server_negotiation.cc:407] Connection header received
0928 23:49:41.279787 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:41.279789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:41.279831 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:41.279932 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:49:41.280180 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.280922 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.282545 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.283163 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.283280 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:41.283374 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:41.283560 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 23:49:41.283638 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:49:41.283758 8382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:41.279108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:42659 (local address 127.6.152.2:52507)
0928 23:49:41.279618 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:41.279631 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:41.279717 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:41.279954 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:41.279956 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:41.279970 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:49:41.280071 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:41.280075 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.281033 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.281039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:41.282399 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:41.282410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.283305 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.283308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:41.283411 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:41.283436 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 23:49:41.283518 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 23:49:41.283592 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":146,"thread_start_us":140,"threads_started":1}
I20260928 23:49:41.283911 8103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:41.279102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:40847 (local address 127.6.152.2:34305)
0928 23:49:41.279208 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:41.279223 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:41.279320 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:41.279622 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:41.279627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:41.279651 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:49:41.279786 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:41.279794 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.280932 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.280934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:41.282567 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:41.282574 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.283465 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.283470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:41.283583 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:41.283600 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 23:49:41.283691 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 23:49:41.283764 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 23:49:41.283892 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:41.279242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:34305 (local address 127.6.152.3:40847)
0928 23:49:41.279369 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:41.279374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:41.279396 (+ 22us) server_negotiation.cc:407] Connection header received
0928 23:49:41.279460 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:41.279463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:41.279504 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:41.279577 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:49:41.279945 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.280758 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.282688 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.283343 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.283443 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:41.283500 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:41.283708 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 23:49:41.283779 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 23:49:41.284570 8290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385b424413e64e6597e247a8b47cc694" candidate_uuid: "3b234a727a7a4bafa8d69622cf3460fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93ea9cdce01f4b73bd502aa818473a43" is_pre_election: true
I20260928 23:49:41.284565 8024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385b424413e64e6597e247a8b47cc694" candidate_uuid: "3b234a727a7a4bafa8d69622cf3460fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4d4ae1ad144d128642390b850e6b89" is_pre_election: true
I20260928 23:49:41.284930 8290 raft_consensus.cc:2454] T 385b424413e64e6597e247a8b47cc694 P 93ea9cdce01f4b73bd502aa818473a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b234a727a7a4bafa8d69622cf3460fb in term 0.
I20260928 23:49:41.284930 8024 raft_consensus.cc:2454] T 385b424413e64e6597e247a8b47cc694 P 1e4d4ae1ad144d128642390b850e6b89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b234a727a7a4bafa8d69622cf3460fb in term 0.
I20260928 23:49:41.285434 8098 leader_election.cc:302] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b234a727a7a4bafa8d69622cf3460fb, 93ea9cdce01f4b73bd502aa818473a43; no voters:
I20260928 23:49:41.285679 8380 raft_consensus.cc:2781] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:49:41.285787 8380 raft_consensus.cc:482] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:49:41.285846 8380 raft_consensus.cc:3037] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:41.286885 8380 raft_consensus.cc:504] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.287281 8380 leader_election.cc:288] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb [CANDIDATE]: Term 1 election: Requested vote from peers 1e4d4ae1ad144d128642390b850e6b89 (127.6.152.1:42659), 93ea9cdce01f4b73bd502aa818473a43 (127.6.152.3:40847)
I20260928 23:49:41.287662 8024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385b424413e64e6597e247a8b47cc694" candidate_uuid: "3b234a727a7a4bafa8d69622cf3460fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4d4ae1ad144d128642390b850e6b89"
I20260928 23:49:41.287663 8290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385b424413e64e6597e247a8b47cc694" candidate_uuid: "3b234a727a7a4bafa8d69622cf3460fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93ea9cdce01f4b73bd502aa818473a43"
I20260928 23:49:41.287796 8290 raft_consensus.cc:3037] T 385b424413e64e6597e247a8b47cc694 P 93ea9cdce01f4b73bd502aa818473a43 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:41.287796 8024 raft_consensus.cc:3037] T 385b424413e64e6597e247a8b47cc694 P 1e4d4ae1ad144d128642390b850e6b89 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:41.289196 8024 raft_consensus.cc:2454] T 385b424413e64e6597e247a8b47cc694 P 1e4d4ae1ad144d128642390b850e6b89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b234a727a7a4bafa8d69622cf3460fb in term 1.
I20260928 23:49:41.289463 8290 raft_consensus.cc:2454] T 385b424413e64e6597e247a8b47cc694 P 93ea9cdce01f4b73bd502aa818473a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b234a727a7a4bafa8d69622cf3460fb in term 1.
I20260928 23:49:41.289543 8095 leader_election.cc:302] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e4d4ae1ad144d128642390b850e6b89, 3b234a727a7a4bafa8d69622cf3460fb; no voters:
I20260928 23:49:41.289727 8380 raft_consensus.cc:2781] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:49:41.289944 8380 raft_consensus.cc:686] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb [term 1 LEADER]: Becoming Leader. State: Replica: 3b234a727a7a4bafa8d69622cf3460fb, State: Running, Role: LEADER
I20260928 23:49:41.290278 8380 consensus_queue.cc:237] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.293587 7867 catalog_manager.cc:5495] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b234a727a7a4bafa8d69622cf3460fb (127.6.152.2). New cstate: current_term: 1 leader_uuid: "3b234a727a7a4bafa8d69622cf3460fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } health_report { overall_health: UNKNOWN } } }
I20260928 23:49:41.318037 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:41.314214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48852 (local address 127.6.152.2:46359)
0928 23:49:41.314379 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:41.314383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:41.314426 (+ 43us) server_negotiation.cc:407] Connection header received
0928 23:49:41.314633 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:41.314636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:41.314689 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:41.314771 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:49:41.314997 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.315526 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.316375 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.316596 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.316684 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:41.317077 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:49:41.317607 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:49:41.317717 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:41.317820 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 23:49:41.317877 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 23:49:41.320117 8351 tablet.cc:2369] T 385b424413e64e6597e247a8b47cc694 P 93ea9cdce01f4b73bd502aa818473a43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:49:41.324252 8290 raft_consensus.cc:1261] T 385b424413e64e6597e247a8b47cc694 P 93ea9cdce01f4b73bd502aa818473a43 [term 1 FOLLOWER]: Refusing update from remote peer 3b234a727a7a4bafa8d69622cf3460fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:49:41.324254 8024 raft_consensus.cc:1261] T 385b424413e64e6597e247a8b47cc694 P 1e4d4ae1ad144d128642390b850e6b89 [term 1 FOLLOWER]: Refusing update from remote peer 3b234a727a7a4bafa8d69622cf3460fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:49:41.324950 8380 consensus_queue.cc:1035] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:49:41.325203 8383 consensus_queue.cc:1035] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:49:41.336216 8388 mvcc.cc:205] Tried to move back new op lower bound from 7334458905896423424 to 7334458905767702528. Current Snapshot: MvccSnapshot[applied={T|T < 7334458905896423424 or (T in {7334458905896423424})}]
I20260928 23:49:41.346271 7867 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:49:41.346458 7867 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:49:41.348991 7867 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4: Sending DeleteTablet for 3 replicas of tablet 385b424413e64e6597e247a8b47cc694
I20260928 23:49:41.349745 8004 tablet_service.cc:1514] Processing DeleteTablet for tablet 385b424413e64e6597e247a8b47cc694 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:49:41 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:36844
I20260928 23:49:41.349825 8284 tablet_service.cc:1514] Processing DeleteTablet for tablet 385b424413e64e6597e247a8b47cc694 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:49:41 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:59116
I20260928 23:49:41.350232 8144 tablet_service.cc:1514] Processing DeleteTablet for tablet 385b424413e64e6597e247a8b47cc694 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:49:41 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:57220
I20260928 23:49:41.350687 8398 tablet_replica.cc:330] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb: stopping tablet replica
I20260928 23:49:41.350970 8398 raft_consensus.cc:2229] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:49:41.350936 7867 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 23:49:41.351295 8398 raft_consensus.cc:2258] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 23:49:41.351323 7867 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:49:41.352638 8398 ts_tablet_manager.cc:1905] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:49:41.355253 8398 ts_tablet_manager.cc:1918] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:49:41.355340 8398 log.cc:1196] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-1/wal/wals/385b424413e64e6597e247a8b47cc694
I20260928 23:49:41.355638 8398 ts_tablet_manager.cc:1939] T 385b424413e64e6597e247a8b47cc694 P 3b234a727a7a4bafa8d69622cf3460fb: Deleting consensus metadata
I20260928 23:49:41.356681 7853 catalog_manager.cc:4841] TS 3b234a727a7a4bafa8d69622cf3460fb (127.6.152.2:46359): tablet 385b424413e64e6597e247a8b47cc694 (table test-table [id=ce668213c76748109be68f435d8efe39]) successfully deleted
I20260928 23:49:41.357558 8004 tablet_service.cc:1467] Processing CreateTablet for tablet 4415a04297e14fbca338449c86351bfa (DEFAULT_TABLE table=test-table [id=0ad431e051a2420d84b59bd2d449f2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:41.357659 8144 tablet_service.cc:1467] Processing CreateTablet for tablet 4415a04297e14fbca338449c86351bfa (DEFAULT_TABLE table=test-table [id=0ad431e051a2420d84b59bd2d449f2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:41.357858 8144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4415a04297e14fbca338449c86351bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:41.357861 8004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4415a04297e14fbca338449c86351bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:41.357929 8284 tablet_service.cc:1467] Processing CreateTablet for tablet 4415a04297e14fbca338449c86351bfa (DEFAULT_TABLE table=test-table [id=0ad431e051a2420d84b59bd2d449f2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:41.358156 8284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4415a04297e14fbca338449c86351bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:41.359977 8373 tablet_bootstrap.cc:492] T 4415a04297e14fbca338449c86351bfa P 93ea9cdce01f4b73bd502aa818473a43: Bootstrap starting.
I20260928 23:49:41.360045 8375 tablet_bootstrap.cc:492] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb: Bootstrap starting.
I20260928 23:49:41.360481 8396 tablet_replica.cc:330] T 385b424413e64e6597e247a8b47cc694 P 93ea9cdce01f4b73bd502aa818473a43: stopping tablet replica
I20260928 23:49:41.360870 8373 tablet_bootstrap.cc:654] T 4415a04297e14fbca338449c86351bfa P 93ea9cdce01f4b73bd502aa818473a43: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:41.360950 8375 tablet_bootstrap.cc:654] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:41.361045 8374 tablet_bootstrap.cc:492] T 4415a04297e14fbca338449c86351bfa P 1e4d4ae1ad144d128642390b850e6b89: Bootstrap starting.
I20260928 23:49:41.362056 8374 tablet_bootstrap.cc:654] T 4415a04297e14fbca338449c86351bfa P 1e4d4ae1ad144d128642390b850e6b89: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:41.362169 8373 tablet_bootstrap.cc:492] T 4415a04297e14fbca338449c86351bfa P 93ea9cdce01f4b73bd502aa818473a43: No bootstrap required, opened a new log
I20260928 23:49:41.362243 8373 ts_tablet_manager.cc:1397] T 4415a04297e14fbca338449c86351bfa P 93ea9cdce01f4b73bd502aa818473a43: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 23:49:41.362421 8375 tablet_bootstrap.cc:492] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb: No bootstrap required, opened a new log
I20260928 23:49:41.362490 8375 ts_tablet_manager.cc:1397] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:49:41.362694 8373 raft_consensus.cc:348] T 4415a04297e14fbca338449c86351bfa P 93ea9cdce01f4b73bd502aa818473a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.362821 8373 raft_consensus.cc:374] T 4415a04297e14fbca338449c86351bfa P 93ea9cdce01f4b73bd502aa818473a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:41.362856 8373 raft_consensus.cc:729] T 4415a04297e14fbca338449c86351bfa P 93ea9cdce01f4b73bd502aa818473a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93ea9cdce01f4b73bd502aa818473a43, State: Initialized, Role: FOLLOWER
I20260928 23:49:41.362973 8373 consensus_queue.cc:260] T 4415a04297e14fbca338449c86351bfa P 93ea9cdce01f4b73bd502aa818473a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.362972 8375 raft_consensus.cc:348] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.363080 8375 raft_consensus.cc:374] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:41.363106 8375 raft_consensus.cc:729] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b234a727a7a4bafa8d69622cf3460fb, State: Initialized, Role: FOLLOWER
I20260928 23:49:41.363224 8373 ts_tablet_manager.cc:1428] T 4415a04297e14fbca338449c86351bfa P 93ea9cdce01f4b73bd502aa818473a43: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 23:49:41.363204 8375 consensus_queue.cc:260] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.363466 8375 ts_tablet_manager.cc:1428] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:49:41.360848 8396 raft_consensus.cc:2229] T 385b424413e64e6597e247a8b47cc694 P 93ea9cdce01f4b73bd502aa818473a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:49:41.363960 8396 raft_consensus.cc:2258] T 385b424413e64e6597e247a8b47cc694 P 93ea9cdce01f4b73bd502aa818473a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:49:41.365518 8397 tablet_replica.cc:330] T 385b424413e64e6597e247a8b47cc694 P 1e4d4ae1ad144d128642390b850e6b89: stopping tablet replica
I20260928 23:49:41.365768 8396 ts_tablet_manager.cc:1905] T 385b424413e64e6597e247a8b47cc694 P 93ea9cdce01f4b73bd502aa818473a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:49:41.365823 8397 raft_consensus.cc:2229] T 385b424413e64e6597e247a8b47cc694 P 1e4d4ae1ad144d128642390b850e6b89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:49:41.366181 8397 raft_consensus.cc:2258] T 385b424413e64e6597e247a8b47cc694 P 1e4d4ae1ad144d128642390b850e6b89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:49:41.367736 8374 tablet_bootstrap.cc:492] T 4415a04297e14fbca338449c86351bfa P 1e4d4ae1ad144d128642390b850e6b89: No bootstrap required, opened a new log
I20260928 23:49:41.367828 8374 ts_tablet_manager.cc:1397] T 4415a04297e14fbca338449c86351bfa P 1e4d4ae1ad144d128642390b850e6b89: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260928 23:49:41.368249 8374 raft_consensus.cc:348] T 4415a04297e14fbca338449c86351bfa P 1e4d4ae1ad144d128642390b850e6b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.368358 8374 raft_consensus.cc:374] T 4415a04297e14fbca338449c86351bfa P 1e4d4ae1ad144d128642390b850e6b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:41.368387 8374 raft_consensus.cc:729] T 4415a04297e14fbca338449c86351bfa P 1e4d4ae1ad144d128642390b850e6b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4d4ae1ad144d128642390b850e6b89, State: Initialized, Role: FOLLOWER
I20260928 23:49:41.368495 8374 consensus_queue.cc:260] T 4415a04297e14fbca338449c86351bfa P 1e4d4ae1ad144d128642390b850e6b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.368736 8374 ts_tablet_manager.cc:1428] T 4415a04297e14fbca338449c86351bfa P 1e4d4ae1ad144d128642390b850e6b89: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:49:41.369385 8397 ts_tablet_manager.cc:1905] T 385b424413e64e6597e247a8b47cc694 P 1e4d4ae1ad144d128642390b850e6b89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:49:41.371834 8397 ts_tablet_manager.cc:1918] T 385b424413e64e6597e247a8b47cc694 P 1e4d4ae1ad144d128642390b850e6b89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:49:41.371943 8397 log.cc:1196] T 385b424413e64e6597e247a8b47cc694 P 1e4d4ae1ad144d128642390b850e6b89: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-0/wal/wals/385b424413e64e6597e247a8b47cc694
I20260928 23:49:41.372227 8397 ts_tablet_manager.cc:1939] T 385b424413e64e6597e247a8b47cc694 P 1e4d4ae1ad144d128642390b850e6b89: Deleting consensus metadata
I20260928 23:49:41.373178 8396 ts_tablet_manager.cc:1918] T 385b424413e64e6597e247a8b47cc694 P 93ea9cdce01f4b73bd502aa818473a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:49:41.373308 8396 log.cc:1196] T 385b424413e64e6597e247a8b47cc694 P 93ea9cdce01f4b73bd502aa818473a43: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/385b424413e64e6597e247a8b47cc694
I20260928 23:49:41.373346 7851 catalog_manager.cc:4841] TS 1e4d4ae1ad144d128642390b850e6b89 (127.6.152.1:42659): tablet 385b424413e64e6597e247a8b47cc694 (table test-table [id=ce668213c76748109be68f435d8efe39]) successfully deleted
I20260928 23:49:41.373603 8396 ts_tablet_manager.cc:1939] T 385b424413e64e6597e247a8b47cc694 P 93ea9cdce01f4b73bd502aa818473a43: Deleting consensus metadata
I20260928 23:49:41.374756 7854 catalog_manager.cc:4841] TS 93ea9cdce01f4b73bd502aa818473a43 (127.6.152.3:40847): tablet 385b424413e64e6597e247a8b47cc694 (table test-table [id=ce668213c76748109be68f435d8efe39]) successfully deleted
I20260928 23:49:41.375533 8393 raft_consensus.cc:482] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:49:41.375658 8393 raft_consensus.cc:504] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.376053 8393 leader_election.cc:288] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e4d4ae1ad144d128642390b850e6b89 (127.6.152.1:42659), 93ea9cdce01f4b73bd502aa818473a43 (127.6.152.3:40847)
I20260928 23:49:41.376423 8290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4415a04297e14fbca338449c86351bfa" candidate_uuid: "3b234a727a7a4bafa8d69622cf3460fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93ea9cdce01f4b73bd502aa818473a43" is_pre_election: true
I20260928 23:49:41.376578 8290 raft_consensus.cc:2454] T 4415a04297e14fbca338449c86351bfa P 93ea9cdce01f4b73bd502aa818473a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b234a727a7a4bafa8d69622cf3460fb in term 0.
I20260928 23:49:41.376555 8024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4415a04297e14fbca338449c86351bfa" candidate_uuid: "3b234a727a7a4bafa8d69622cf3460fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4d4ae1ad144d128642390b850e6b89" is_pre_election: true
I20260928 23:49:41.376678 8024 raft_consensus.cc:2454] T 4415a04297e14fbca338449c86351bfa P 1e4d4ae1ad144d128642390b850e6b89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b234a727a7a4bafa8d69622cf3460fb in term 0.
I20260928 23:49:41.376912 8098 leader_election.cc:302] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b234a727a7a4bafa8d69622cf3460fb, 93ea9cdce01f4b73bd502aa818473a43; no voters:
I20260928 23:49:41.377122 8393 raft_consensus.cc:2781] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:49:41.377177 8393 raft_consensus.cc:482] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:49:41.377215 8393 raft_consensus.cc:3037] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:41.378342 8393 raft_consensus.cc:504] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.378684 8393 leader_election.cc:288] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb [CANDIDATE]: Term 1 election: Requested vote from peers 1e4d4ae1ad144d128642390b850e6b89 (127.6.152.1:42659), 93ea9cdce01f4b73bd502aa818473a43 (127.6.152.3:40847)
W20260928 23:49:41.378748 8071 tablet.cc:2369] T 4415a04297e14fbca338449c86351bfa P 1e4d4ae1ad144d128642390b850e6b89: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:49:41.379001 8290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4415a04297e14fbca338449c86351bfa" candidate_uuid: "3b234a727a7a4bafa8d69622cf3460fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93ea9cdce01f4b73bd502aa818473a43"
I20260928 23:49:41.379117 8290 raft_consensus.cc:3037] T 4415a04297e14fbca338449c86351bfa P 93ea9cdce01f4b73bd502aa818473a43 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:41.379540 8024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4415a04297e14fbca338449c86351bfa" candidate_uuid: "3b234a727a7a4bafa8d69622cf3460fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4d4ae1ad144d128642390b850e6b89"
I20260928 23:49:41.379650 8024 raft_consensus.cc:3037] T 4415a04297e14fbca338449c86351bfa P 1e4d4ae1ad144d128642390b850e6b89 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:41.380244 8290 raft_consensus.cc:2454] T 4415a04297e14fbca338449c86351bfa P 93ea9cdce01f4b73bd502aa818473a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b234a727a7a4bafa8d69622cf3460fb in term 1.
I20260928 23:49:41.380503 8098 leader_election.cc:302] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b234a727a7a4bafa8d69622cf3460fb, 93ea9cdce01f4b73bd502aa818473a43; no voters:
I20260928 23:49:41.380683 8393 raft_consensus.cc:2781] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:49:41.380757 8024 raft_consensus.cc:2454] T 4415a04297e14fbca338449c86351bfa P 1e4d4ae1ad144d128642390b850e6b89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b234a727a7a4bafa8d69622cf3460fb in term 1.
I20260928 23:49:41.380776 8393 raft_consensus.cc:686] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb [term 1 LEADER]: Becoming Leader. State: Replica: 3b234a727a7a4bafa8d69622cf3460fb, State: Running, Role: LEADER
I20260928 23:49:41.380971 8393 consensus_queue.cc:237] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.382526 7867 catalog_manager.cc:5495] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b234a727a7a4bafa8d69622cf3460fb (127.6.152.2). New cstate: current_term: 1 leader_uuid: "3b234a727a7a4bafa8d69622cf3460fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } health_report { overall_health: UNKNOWN } } }
Sep 28 23:49:41 dist-test-slave-p5bw krb5kdc[7822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639380, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:41.423043 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:41.414607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:34703 (local address 127.0.0.1:43744)
0928 23:49:41.414939 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:41.414955 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:41.415077 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:41.415367 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:41.415370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:41.415799 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:41.415886 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:41.415892 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.416635 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.416641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:41.417273 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:41.417281 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.417713 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.417717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:41.417795 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:41.418389 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:41.418400 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:41.419751 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:41.421445 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:41.421456 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:41.421464 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:41.421698 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:41.421973 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:41.421976 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:41.421981 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:41.422066 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:41.422439 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:41.422447 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:41.422563 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 23:49:41.422679 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 23:49:41.422847 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":107,"threads_started":1}
I20260928 23:49:41.423043 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:41.414681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43744 (local address 127.6.152.62:34703)
0928 23:49:41.414820 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:41.414824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:41.415036 (+ 212us) server_negotiation.cc:407] Connection header received
0928 23:49:41.415197 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:41.415199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:41.415240 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:41.415351 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:41.416021 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.416527 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.417394 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.417604 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.417713 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:41.419897 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:41.419900 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:41.419902 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:41.419921 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:41.421336 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:41.421799 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:41.421805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:41.421807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:41.421881 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:41.422159 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:41.422162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:41.422164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:41.422326 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:41.422429 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:41.422724 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 23:49:41.422916 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":61}
I20260928 23:49:41.426863 7867 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:43744:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 23:49:41.427652 7867 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:49:41.432570 8004 tablet_service.cc:1467] Processing CreateTablet for tablet e33cb08af1134fbb8d0cbaa521d52fba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f42107f1b8974b0c94a6e467bdaeea5e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:49:41.432715 8284 tablet_service.cc:1467] Processing CreateTablet for tablet e33cb08af1134fbb8d0cbaa521d52fba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f42107f1b8974b0c94a6e467bdaeea5e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:49:41.432721 8144 tablet_service.cc:1467] Processing CreateTablet for tablet e33cb08af1134fbb8d0cbaa521d52fba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f42107f1b8974b0c94a6e467bdaeea5e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:49:41.433043 8004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e33cb08af1134fbb8d0cbaa521d52fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:41.433096 8284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e33cb08af1134fbb8d0cbaa521d52fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:41.433096 8144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e33cb08af1134fbb8d0cbaa521d52fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:41.435694 8374 tablet_bootstrap.cc:492] T e33cb08af1134fbb8d0cbaa521d52fba P 1e4d4ae1ad144d128642390b850e6b89: Bootstrap starting.
I20260928 23:49:41.435755 8373 tablet_bootstrap.cc:492] T e33cb08af1134fbb8d0cbaa521d52fba P 93ea9cdce01f4b73bd502aa818473a43: Bootstrap starting.
I20260928 23:49:41.436553 8374 tablet_bootstrap.cc:654] T e33cb08af1134fbb8d0cbaa521d52fba P 1e4d4ae1ad144d128642390b850e6b89: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:41.436642 8373 tablet_bootstrap.cc:654] T e33cb08af1134fbb8d0cbaa521d52fba P 93ea9cdce01f4b73bd502aa818473a43: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:41.437390 8375 tablet_bootstrap.cc:492] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb: Bootstrap starting.
I20260928 23:49:41.438179 8375 tablet_bootstrap.cc:654] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:41.438267 8373 tablet_bootstrap.cc:492] T e33cb08af1134fbb8d0cbaa521d52fba P 93ea9cdce01f4b73bd502aa818473a43: No bootstrap required, opened a new log
I20260928 23:49:41.438350 8373 ts_tablet_manager.cc:1397] T e33cb08af1134fbb8d0cbaa521d52fba P 93ea9cdce01f4b73bd502aa818473a43: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 23:49:41.438766 8373 raft_consensus.cc:348] T e33cb08af1134fbb8d0cbaa521d52fba P 93ea9cdce01f4b73bd502aa818473a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.438894 8373 raft_consensus.cc:374] T e33cb08af1134fbb8d0cbaa521d52fba P 93ea9cdce01f4b73bd502aa818473a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:41.438931 8373 raft_consensus.cc:729] T e33cb08af1134fbb8d0cbaa521d52fba P 93ea9cdce01f4b73bd502aa818473a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93ea9cdce01f4b73bd502aa818473a43, State: Initialized, Role: FOLLOWER
I20260928 23:49:41.439046 8373 consensus_queue.cc:260] T e33cb08af1134fbb8d0cbaa521d52fba P 93ea9cdce01f4b73bd502aa818473a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.439111 8374 tablet_bootstrap.cc:492] T e33cb08af1134fbb8d0cbaa521d52fba P 1e4d4ae1ad144d128642390b850e6b89: No bootstrap required, opened a new log
I20260928 23:49:41.439193 8374 ts_tablet_manager.cc:1397] T e33cb08af1134fbb8d0cbaa521d52fba P 1e4d4ae1ad144d128642390b850e6b89: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 23:49:41.439284 8373 ts_tablet_manager.cc:1428] T e33cb08af1134fbb8d0cbaa521d52fba P 93ea9cdce01f4b73bd502aa818473a43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:49:41.439440 8375 tablet_bootstrap.cc:492] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb: No bootstrap required, opened a new log
I20260928 23:49:41.439512 8375 ts_tablet_manager.cc:1397] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:49:41.439513 8381 tablet_replica.cc:439] T e33cb08af1134fbb8d0cbaa521d52fba P 93ea9cdce01f4b73bd502aa818473a43: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } } }
I20260928 23:49:41.439634 8381 tablet_replica.cc:442] T e33cb08af1134fbb8d0cbaa521d52fba P 93ea9cdce01f4b73bd502aa818473a43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:49:41.439644 8374 raft_consensus.cc:348] T e33cb08af1134fbb8d0cbaa521d52fba P 1e4d4ae1ad144d128642390b850e6b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.439742 8374 raft_consensus.cc:374] T e33cb08af1134fbb8d0cbaa521d52fba P 1e4d4ae1ad144d128642390b850e6b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:41.439775 8374 raft_consensus.cc:729] T e33cb08af1134fbb8d0cbaa521d52fba P 1e4d4ae1ad144d128642390b850e6b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4d4ae1ad144d128642390b850e6b89, State: Initialized, Role: FOLLOWER
I20260928 23:49:41.439884 8374 consensus_queue.cc:260] T e33cb08af1134fbb8d0cbaa521d52fba P 1e4d4ae1ad144d128642390b850e6b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.439924 8375 raft_consensus.cc:348] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.440045 8375 raft_consensus.cc:374] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:41.440080 8375 raft_consensus.cc:729] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b234a727a7a4bafa8d69622cf3460fb, State: Initialized, Role: FOLLOWER
I20260928 23:49:41.440160 8379 tablet_replica.cc:439] T e33cb08af1134fbb8d0cbaa521d52fba P 1e4d4ae1ad144d128642390b850e6b89: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } } }
I20260928 23:49:41.440196 8375 consensus_queue.cc:260] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.440291 8379 tablet_replica.cc:442] T e33cb08af1134fbb8d0cbaa521d52fba P 1e4d4ae1ad144d128642390b850e6b89: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:49:41.440451 8375 ts_tablet_manager.cc:1428] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:49:41.440652 8393 tablet_replica.cc:439] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } } }
I20260928 23:49:41.440770 8393 tablet_replica.cc:442] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:49:41.440165 8374 ts_tablet_manager.cc:1428] T e33cb08af1134fbb8d0cbaa521d52fba P 1e4d4ae1ad144d128642390b850e6b89: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 23:49:41.551147 8393 raft_consensus.cc:482] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:49:41.551283 8393 raft_consensus.cc:504] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.551741 8393 leader_election.cc:288] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e4d4ae1ad144d128642390b850e6b89 (127.6.152.1:42659), 93ea9cdce01f4b73bd502aa818473a43 (127.6.152.3:40847)
I20260928 23:49:41.552227 8024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33cb08af1134fbb8d0cbaa521d52fba" candidate_uuid: "3b234a727a7a4bafa8d69622cf3460fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4d4ae1ad144d128642390b850e6b89" is_pre_election: true
I20260928 23:49:41.552227 8290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33cb08af1134fbb8d0cbaa521d52fba" candidate_uuid: "3b234a727a7a4bafa8d69622cf3460fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93ea9cdce01f4b73bd502aa818473a43" is_pre_election: true
I20260928 23:49:41.552380 8290 raft_consensus.cc:2454] T e33cb08af1134fbb8d0cbaa521d52fba P 93ea9cdce01f4b73bd502aa818473a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b234a727a7a4bafa8d69622cf3460fb in term 0.
I20260928 23:49:41.552386 8024 raft_consensus.cc:2454] T e33cb08af1134fbb8d0cbaa521d52fba P 1e4d4ae1ad144d128642390b850e6b89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b234a727a7a4bafa8d69622cf3460fb in term 0.
I20260928 23:49:41.552673 8098 leader_election.cc:302] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b234a727a7a4bafa8d69622cf3460fb, 93ea9cdce01f4b73bd502aa818473a43; no voters:
I20260928 23:49:41.552904 8393 raft_consensus.cc:2781] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:49:41.552968 8393 raft_consensus.cc:482] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:49:41.552997 8393 raft_consensus.cc:3037] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:41.554297 8393 raft_consensus.cc:504] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.554673 8393 leader_election.cc:288] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb [CANDIDATE]: Term 1 election: Requested vote from peers 1e4d4ae1ad144d128642390b850e6b89 (127.6.152.1:42659), 93ea9cdce01f4b73bd502aa818473a43 (127.6.152.3:40847)
I20260928 23:49:41.555020 8024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33cb08af1134fbb8d0cbaa521d52fba" candidate_uuid: "3b234a727a7a4bafa8d69622cf3460fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4d4ae1ad144d128642390b850e6b89"
I20260928 23:49:41.555040 8290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e33cb08af1134fbb8d0cbaa521d52fba" candidate_uuid: "3b234a727a7a4bafa8d69622cf3460fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93ea9cdce01f4b73bd502aa818473a43"
I20260928 23:49:41.555130 8024 raft_consensus.cc:3037] T e33cb08af1134fbb8d0cbaa521d52fba P 1e4d4ae1ad144d128642390b850e6b89 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:41.555169 8290 raft_consensus.cc:3037] T e33cb08af1134fbb8d0cbaa521d52fba P 93ea9cdce01f4b73bd502aa818473a43 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:41.555986 8024 raft_consensus.cc:2454] T e33cb08af1134fbb8d0cbaa521d52fba P 1e4d4ae1ad144d128642390b850e6b89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b234a727a7a4bafa8d69622cf3460fb in term 1.
I20260928 23:49:41.556089 8290 raft_consensus.cc:2454] T e33cb08af1134fbb8d0cbaa521d52fba P 93ea9cdce01f4b73bd502aa818473a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b234a727a7a4bafa8d69622cf3460fb in term 1.
I20260928 23:49:41.556303 8095 leader_election.cc:302] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e4d4ae1ad144d128642390b850e6b89, 3b234a727a7a4bafa8d69622cf3460fb; no voters:
I20260928 23:49:41.556501 8393 raft_consensus.cc:2781] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:49:41.556610 8393 raft_consensus.cc:686] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb [term 1 LEADER]: Becoming Leader. State: Replica: 3b234a727a7a4bafa8d69622cf3460fb, State: Running, Role: LEADER
I20260928 23:49:41.556742 8393 consensus_queue.cc:237] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } }
I20260928 23:49:41.557354 8383 tablet_replica.cc:439] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb: TxnStatusTablet state changed. Reason: New leader 3b234a727a7a4bafa8d69622cf3460fb. Latest consensus state: current_term: 1 leader_uuid: "3b234a727a7a4bafa8d69622cf3460fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } } }
I20260928 23:49:41.557482 8383 tablet_replica.cc:442] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:49:41.557756 8413 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 23:49:41.557929 8413 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 23:49:41.558465 7866 catalog_manager.cc:5495] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b234a727a7a4bafa8d69622cf3460fb (127.6.152.2). New cstate: current_term: 1 leader_uuid: "3b234a727a7a4bafa8d69622cf3460fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } health_report { overall_health: UNKNOWN } } }
I20260928 23:49:41.593070 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:41.589503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:46359 (local address 127.0.0.1:48856)
0928 23:49:41.589631 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:41.589647 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:41.589747 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:41.590032 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:41.590034 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:41.590042 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:49:41.590133 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:41.590138 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.591125 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.591128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:41.591728 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:41.591736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.592269 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.592271 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:41.592334 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:41.592389 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:49:41.592711 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:49:41.592728 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 23:49:41.592808 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 23:49:41.592932 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 23:49:41.593070 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:41.589596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48856 (local address 127.6.152.2:46359)
0928 23:49:41.589718 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:41.589722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:41.589738 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:49:41.589860 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:41.589863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:41.589914 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:41.589996 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:49:41.590229 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.591025 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.591832 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.592164 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.592266 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:41.592550 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:49:41.592628 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:49:41.592708 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:41.592899 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 23:49:41.592945 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 23:49:41.595369 8211 tablet.cc:2369] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:49:41.595613 8024 raft_consensus.cc:1261] T e33cb08af1134fbb8d0cbaa521d52fba P 1e4d4ae1ad144d128642390b850e6b89 [term 1 FOLLOWER]: Refusing update from remote peer 3b234a727a7a4bafa8d69622cf3460fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:49:41.596089 8383 consensus_queue.cc:1035] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:49:41.596194 8290 raft_consensus.cc:1261] T e33cb08af1134fbb8d0cbaa521d52fba P 93ea9cdce01f4b73bd502aa818473a43 [term 1 FOLLOWER]: Refusing update from remote peer 3b234a727a7a4bafa8d69622cf3460fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:49:41.596587 8383 consensus_queue.cc:1035] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:49:41.597786 8381 tablet_replica.cc:439] T e33cb08af1134fbb8d0cbaa521d52fba P 93ea9cdce01f4b73bd502aa818473a43: TxnStatusTablet state changed. Reason: New leader 3b234a727a7a4bafa8d69622cf3460fb. Latest consensus state: current_term: 1 leader_uuid: "3b234a727a7a4bafa8d69622cf3460fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } } }
I20260928 23:49:41.597898 8381 tablet_replica.cc:442] T e33cb08af1134fbb8d0cbaa521d52fba P 93ea9cdce01f4b73bd502aa818473a43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:49:41.597945 8379 tablet_replica.cc:439] T e33cb08af1134fbb8d0cbaa521d52fba P 1e4d4ae1ad144d128642390b850e6b89: TxnStatusTablet state changed. Reason: New leader 3b234a727a7a4bafa8d69622cf3460fb. Latest consensus state: current_term: 1 leader_uuid: "3b234a727a7a4bafa8d69622cf3460fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } } }
I20260928 23:49:41.598048 8379 tablet_replica.cc:442] T e33cb08af1134fbb8d0cbaa521d52fba P 1e4d4ae1ad144d128642390b850e6b89: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:49:41.599192 8380 tablet_replica.cc:439] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b234a727a7a4bafa8d69622cf3460fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } } }
I20260928 23:49:41.599299 8380 tablet_replica.cc:442] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:49:41.599697 8386 mvcc.cc:205] Tried to move back new op lower bound from 7334458907011420160 to 7334458906857672704. Current Snapshot: MvccSnapshot[applied={T|T < 7334458907011420160 or (T in {7334458907011420160})}]
I20260928 23:49:41.600402 8387 mvcc.cc:205] Tried to move back new op lower bound from 7334458907011420160 to 7334458906857672704. Current Snapshot: MvccSnapshot[applied={T|T < 7334458907011420160}]
I20260928 23:49:41.600565 8379 tablet_replica.cc:439] T e33cb08af1134fbb8d0cbaa521d52fba P 1e4d4ae1ad144d128642390b850e6b89: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b234a727a7a4bafa8d69622cf3460fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } } }
I20260928 23:49:41.600664 8379 tablet_replica.cc:442] T e33cb08af1134fbb8d0cbaa521d52fba P 1e4d4ae1ad144d128642390b850e6b89: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:49:41.601511 8393 tablet_replica.cc:439] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b234a727a7a4bafa8d69622cf3460fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } } }
I20260928 23:49:41.601625 8393 tablet_replica.cc:442] T e33cb08af1134fbb8d0cbaa521d52fba P 3b234a727a7a4bafa8d69622cf3460fb: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:49:41.602542 8381 tablet_replica.cc:439] T e33cb08af1134fbb8d0cbaa521d52fba P 93ea9cdce01f4b73bd502aa818473a43: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b234a727a7a4bafa8d69622cf3460fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 } } peers { permanent_uuid: "3b234a727a7a4bafa8d69622cf3460fb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46359 } } peers { permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 } } }
I20260928 23:49:41.602648 8381 tablet_replica.cc:442] T e33cb08af1134fbb8d0cbaa521d52fba P 93ea9cdce01f4b73bd502aa818473a43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:49:41.616580 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:41.612960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:47893 (local address 127.6.152.2:46359)
0928 23:49:41.613076 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:41.613080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:41.613358 (+ 278us) server_negotiation.cc:407] Connection header received
0928 23:49:41.613450 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:41.613452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:41.613496 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:41.613614 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:49:41.613858 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.614435 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.615629 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.615829 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.615914 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:41.616193 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:49:41.616256 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:49:41.616306 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:41.616419 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 23:49:41.616457 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 23:49:41.616669 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:41.612952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:46359 (local address 127.6.152.2:47893)
0928 23:49:41.613239 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:41.613253 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:41.613355 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:41.613634 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:41.613637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:41.613644 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:49:41.613748 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:41.613753 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.614533 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.614536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:41.615518 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:41.615529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.615938 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.615941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:41.616027 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:41.616075 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:49:41.616329 (+ 254us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:49:41.616335 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 23:49:41.616398 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 23:49:41.616473 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":95,"threads_started":1}
I20260928 23:49:41.622577 8024 raft_consensus.cc:1261] T 4415a04297e14fbca338449c86351bfa P 1e4d4ae1ad144d128642390b850e6b89 [term 1 FOLLOWER]: Refusing update from remote peer 3b234a727a7a4bafa8d69622cf3460fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:49:41.622912 8290 raft_consensus.cc:1261] T 4415a04297e14fbca338449c86351bfa P 93ea9cdce01f4b73bd502aa818473a43 [term 1 FOLLOWER]: Refusing update from remote peer 3b234a727a7a4bafa8d69622cf3460fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:49:41.622951 8383 consensus_queue.cc:1035] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e4d4ae1ad144d128642390b850e6b89" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:49:41.623287 8383 consensus_queue.cc:1035] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "93ea9cdce01f4b73bd502aa818473a43" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 40847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:49:41.662616 7865 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:49:41.662758 7865 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:49:41.664698 7865 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1d002a27b574ffd969ddfcb9fd58ca4: Sending DeleteTablet for 3 replicas of tablet 4415a04297e14fbca338449c86351bfa
I20260928 23:49:41.665266 8004 tablet_service.cc:1514] Processing DeleteTablet for tablet 4415a04297e14fbca338449c86351bfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:49:41 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:36844
I20260928 23:49:41.665479 8397 tablet_replica.cc:330] T 4415a04297e14fbca338449c86351bfa P 1e4d4ae1ad144d128642390b850e6b89: stopping tablet replica
I20260928 23:49:41.665455 8284 tablet_service.cc:1514] Processing DeleteTablet for tablet 4415a04297e14fbca338449c86351bfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:49:41 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:59116
I20260928 23:49:41.665592 8397 raft_consensus.cc:2229] T 4415a04297e14fbca338449c86351bfa P 1e4d4ae1ad144d128642390b850e6b89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:49:41.665623 8396 tablet_replica.cc:330] T 4415a04297e14fbca338449c86351bfa P 93ea9cdce01f4b73bd502aa818473a43: stopping tablet replica
I20260928 23:49:41.665723 8397 raft_consensus.cc:2258] T 4415a04297e14fbca338449c86351bfa P 1e4d4ae1ad144d128642390b850e6b89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:49:41.665727 8396 raft_consensus.cc:2229] T 4415a04297e14fbca338449c86351bfa P 93ea9cdce01f4b73bd502aa818473a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:49:41.665825 8396 raft_consensus.cc:2258] T 4415a04297e14fbca338449c86351bfa P 93ea9cdce01f4b73bd502aa818473a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:49:41.666620 8144 tablet_service.cc:1514] Processing DeleteTablet for tablet 4415a04297e14fbca338449c86351bfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:49:41 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:57220
I20260928 23:49:41.666663 8396 ts_tablet_manager.cc:1905] T 4415a04297e14fbca338449c86351bfa P 93ea9cdce01f4b73bd502aa818473a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:49:41.666761 8398 tablet_replica.cc:330] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb: stopping tablet replica
I20260928 23:49:41.666851 8398 raft_consensus.cc:2229] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:49:41.666998 8398 raft_consensus.cc:2258] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:49:41.667739 8398 ts_tablet_manager.cc:1905] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:49:41.669389 8396 ts_tablet_manager.cc:1918] T 4415a04297e14fbca338449c86351bfa P 93ea9cdce01f4b73bd502aa818473a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:49:41.669476 8396 log.cc:1196] T 4415a04297e14fbca338449c86351bfa P 93ea9cdce01f4b73bd502aa818473a43: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/4415a04297e14fbca338449c86351bfa
I20260928 23:49:41.669744 8396 ts_tablet_manager.cc:1939] T 4415a04297e14fbca338449c86351bfa P 93ea9cdce01f4b73bd502aa818473a43: Deleting consensus metadata
I20260928 23:49:41.670410 7854 catalog_manager.cc:4841] TS 93ea9cdce01f4b73bd502aa818473a43 (127.6.152.3:40847): tablet 4415a04297e14fbca338449c86351bfa (table test-table [id=0ad431e051a2420d84b59bd2d449f2e7]) successfully deleted
I20260928 23:49:41.670461 8398 ts_tablet_manager.cc:1918] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:49:41.670524 8398 log.cc:1196] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-1/wal/wals/4415a04297e14fbca338449c86351bfa
I20260928 23:49:41.670770 8398 ts_tablet_manager.cc:1939] T 4415a04297e14fbca338449c86351bfa P 3b234a727a7a4bafa8d69622cf3460fb: Deleting consensus metadata
I20260928 23:49:41.671409 7853 catalog_manager.cc:4841] TS 3b234a727a7a4bafa8d69622cf3460fb (127.6.152.2:46359): tablet 4415a04297e14fbca338449c86351bfa (table test-table [id=0ad431e051a2420d84b59bd2d449f2e7]) successfully deleted
I20260928 23:49:41.671710 8397 ts_tablet_manager.cc:1905] T 4415a04297e14fbca338449c86351bfa P 1e4d4ae1ad144d128642390b850e6b89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:49:41.674230 8397 ts_tablet_manager.cc:1918] T 4415a04297e14fbca338449c86351bfa P 1e4d4ae1ad144d128642390b850e6b89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:49:41.674309 8397 log.cc:1196] T 4415a04297e14fbca338449c86351bfa P 1e4d4ae1ad144d128642390b850e6b89: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639374546653-6752-0/minicluster-data/ts-0/wal/wals/4415a04297e14fbca338449c86351bfa
I20260928 23:49:41.674546 8397 ts_tablet_manager.cc:1939] T 4415a04297e14fbca338449c86351bfa P 1e4d4ae1ad144d128642390b850e6b89: Deleting consensus metadata
I20260928 23:49:41.675177 7851 catalog_manager.cc:4841] TS 1e4d4ae1ad144d128642390b850e6b89 (127.6.152.1:42659): tablet 4415a04297e14fbca338449c86351bfa (table test-table [id=0ad431e051a2420d84b59bd2d449f2e7]) successfully deleted
Sep 28 23:49:41 dist-test-slave-p5bw krb5kdc[7822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639381, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.152.1@KRBTEST.COM
I20260928 23:49:41.680233 8369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:41.669016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58218 (local address 127.6.152.1:42659)
0928 23:49:41.669155 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:41.669161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:41.669450 (+ 289us) server_negotiation.cc:407] Connection header received
0928 23:49:41.669784 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:41.669788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:41.669857 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:41.669916 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:41.671061 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.671727 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.672569 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.672799 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.672958 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:41.676725 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:41.676734 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:41.676746 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:41.676768 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:41.678593 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:41.679068 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:41.679073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:41.679074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:41.679146 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:41.679412 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:41.679414 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:41.679416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:41.679628 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:41.679778 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:41.679983 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 23:49:41.680070 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 23:49:41.680712 8064 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58218
I20260928 23:49:41.690198 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:41.682431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43756 (local address 127.6.152.62:34703)
0928 23:49:41.682553 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:41.682558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:41.682635 (+ 77us) server_negotiation.cc:407] Connection header received
0928 23:49:41.682707 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:41.682710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:41.682763 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:41.682872 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:41.683502 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.684309 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.685173 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:41.685501 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:41.685618 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:41.686862 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:41.686866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:41.686869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:41.686897 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:41.688442 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:41.688906 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:41.688910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:41.688912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:41.688999 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:41.689313 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:41.689316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:41.689318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:41.689542 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:41.689678 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:41.689905 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 23:49:41.690037 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 23:49:41.690593 7865 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43756
I20260928 23:49:41.691823 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 7939
I20260928 23:49:41.699133 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 8079
I20260928 23:49:41.706413 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 8219
I20260928 23:49:41.711826 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 7835
2026-09-28T23:49:41Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2700 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:49:41 dist-test-slave-p5bw krb5kdc[8431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:49:41 dist-test-slave-p5bw krb5kdc[8431](info): set up 2 sockets
Sep 28 23:49:41 dist-test-slave-p5bw krb5kdc[8431](info): commencing operation
krb5kdc: starting...
W20260928 23:49:42.773313 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:49:42 dist-test-slave-p5bw krb5kdc[8431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:49:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:49:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:49:42.887912 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:34125
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:34125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:42.994997 8444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:42.995266 8444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:42.995335 8444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:42.999300 8444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:49:42.999369 8444 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 23:49:42.999404 8444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:42.999473 8444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:42.999511 8444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:49:42.999541 8444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:49:43.005131 8444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:34125
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:34125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:43.006103 8444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:43.007061 8444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:49:43.015268 8444 server_base.cc:1034] running on GCE node
W20260928 23:49:43.015254 8449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:43.015410 8452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:43.015256 8450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:43.016213 8444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:43.017266 8444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:43.018429 8444 hybrid_clock.cc:648] HybridClock initialized: now 1790639383018418 us; error 23 us; skew 500 ppm
Sep 28 23:49:43 dist-test-slave-p5bw krb5kdc[8431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639383, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:43.021389 8444 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:49:43.022342 8444 webserver.cc:460] Webserver started at http://127.6.152.62:37613/ using document root <none> and password file <none>
I20260928 23:49:43.022892 8444 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:43.022964 8444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:43.023146 8444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:43.025272 8444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "5a917c45eee74d01830e12a3fc1b8f6c"
format_stamp: "Formatted at 2026-09-28 23:49:43 on dist-test-slave-p5bw"
server_key: "55d705fb52bde6919967f2a6480fd1b7"
server_key_iv: "ed5129126e8f0189316e850615efeb9f"
server_key_version: "encryptionkey@0"
I20260928 23:49:43.025816 8444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "5a917c45eee74d01830e12a3fc1b8f6c"
format_stamp: "Formatted at 2026-09-28 23:49:43 on dist-test-slave-p5bw"
server_key: "55d705fb52bde6919967f2a6480fd1b7"
server_key_iv: "ed5129126e8f0189316e850615efeb9f"
server_key_version: "encryptionkey@0"
I20260928 23:49:43.029587 8444 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 23:49:43.031967 8459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:43.033288 8444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:49:43.033432 8444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "5a917c45eee74d01830e12a3fc1b8f6c"
format_stamp: "Formatted at 2026-09-28 23:49:43 on dist-test-slave-p5bw"
server_key: "55d705fb52bde6919967f2a6480fd1b7"
server_key_iv: "ed5129126e8f0189316e850615efeb9f"
server_key_version: "encryptionkey@0"
I20260928 23:49:43.033552 8444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:43.060154 8444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:43.063453 8444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:43.063681 8444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:43.071605 8444 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:34125
I20260928 23:49:43.071617 8521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:34125 every 8 connection(s)
I20260928 23:49:43.072578 8444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:49:43.073410 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 8444
I20260928 23:49:43.073527 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:49:43.073827 6752 external_mini_cluster.cc:1428] Setting key 7ffd2fd17897ccbbb34dd88c6225fb9d
I20260928 23:49:43.075964 8522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:43.081360 8522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c: Bootstrap starting.
Sep 28 23:49:43 dist-test-slave-p5bw krb5kdc[8431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:43.083575 8522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:43.084770 8522 log.cc:824] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c: Log is configured to *not* fsync() on all Append() calls
I20260928 23:49:43.086910 8522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c: No bootstrap required, opened a new log
I20260928 23:49:43.087934 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.075269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47658 (local address 127.6.152.62:34125)
0928 23:49:43.075745 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:43.075763 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:43.076095 (+ 332us) server_negotiation.cc:407] Connection header received
0928 23:49:43.076893 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:43.076921 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:43.077252 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:43.077587 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:43.078365 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.079354 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.080222 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.080699 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.080894 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.083357 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:43.083362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:43.083375 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:43.083402 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:43.085246 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:43.085734 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:43.085738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:43.085742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:43.085848 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:43.086123 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:43.086125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:43.086126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:43.086482 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:43.086640 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:43.086924 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 23:49:43.087251 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":141,"threads_started":1}
I20260928 23:49:43.090265 8522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a917c45eee74d01830e12a3fc1b8f6c" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 34125 } }
I20260928 23:49:43.090472 8522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:43.090547 8522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a917c45eee74d01830e12a3fc1b8f6c, State: Initialized, Role: FOLLOWER
I20260928 23:49:43.090924 8522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a917c45eee74d01830e12a3fc1b8f6c" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 34125 } }
I20260928 23:49:43.091056 8522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:49:43.091117 8522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:49:43.091210 8522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:43.092041 8522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a917c45eee74d01830e12a3fc1b8f6c" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 34125 } }
I20260928 23:49:43.092314 8522 leader_election.cc:302] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a917c45eee74d01830e12a3fc1b8f6c; no voters:
I20260928 23:49:43.092641 8522 leader_election.cc:288] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:49:43.092752 8527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:49:43.092994 8527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c [term 1 LEADER]: Becoming Leader. State: Replica: 5a917c45eee74d01830e12a3fc1b8f6c, State: Running, Role: LEADER
I20260928 23:49:43.093314 8527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a917c45eee74d01830e12a3fc1b8f6c" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 34125 } }
I20260928 23:49:43.093829 8522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:49:43.095018 8529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a917c45eee74d01830e12a3fc1b8f6c. Latest consensus state: current_term: 1 leader_uuid: "5a917c45eee74d01830e12a3fc1b8f6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a917c45eee74d01830e12a3fc1b8f6c" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 34125 } } }
I20260928 23:49:43.095018 8528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a917c45eee74d01830e12a3fc1b8f6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a917c45eee74d01830e12a3fc1b8f6c" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 34125 } } }
I20260928 23:49:43.095198 8528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c [sys.catalog]: This master's current role is: LEADER
I20260928 23:49:43.095485 8529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c [sys.catalog]: This master's current role is: LEADER
W20260928 23:49:43.098481 8543 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:49:43.098569 8543 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:49:43.098762 8537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:49:43.099416 8537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:49:43.104604 8537 catalog_manager.cc:1347] Generated new cluster ID: 12cf674939394b809d7f1e6b27cffa34
I20260928 23:49:43.104669 8537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:49:43.113217 8537 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:49:43.114069 8537 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:49:43.120975 8537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c: Generated new TSK 0
I20260928 23:49:43.121618 8537 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
I20260928 23:49:43.151433 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:34125
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:43.258591 8548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:43.258855 8548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:43.258947 8548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:43.262933 8548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:43.263008 8548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:43.263078 8548 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 23:49:43.263120 8548 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:49:43.263181 8548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:49:43.268158 8548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.152.62:34125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:43.269174 8548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:43.270136 8548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:43.279071 8554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:43.279067 8553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:43.279088 8556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:43.279275 8548 server_base.cc:1034] running on GCE node
I20260928 23:49:43.279733 8548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:43.280328 8548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:43.281486 8548 hybrid_clock.cc:648] HybridClock initialized: now 1790639383281472 us; error 24 us; skew 500 ppm
Sep 28 23:49:43 dist-test-slave-p5bw krb5kdc[8431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639383, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:43.284066 8548 init.cc:377] Logged in from keytab as kudu/127.6.152.1@KRBTEST.COM (short username kudu)
I20260928 23:49:43.284850 8548 webserver.cc:460] Webserver started at http://127.6.152.1:39193/ using document root <none> and password file <none>
I20260928 23:49:43.285367 8548 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:43.285423 8548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:43.285579 8548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:43.287504 8548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "dbc13a0a54194dbd89fa635b387ddb4e"
format_stamp: "Formatted at 2026-09-28 23:49:43 on dist-test-slave-p5bw"
server_key: "87b0b1ee43efccf5981b67d2c81c9e04"
server_key_iv: "03e72960bba1283c77581f853f5a5039"
server_key_version: "encryptionkey@0"
I20260928 23:49:43.287987 8548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "dbc13a0a54194dbd89fa635b387ddb4e"
format_stamp: "Formatted at 2026-09-28 23:49:43 on dist-test-slave-p5bw"
server_key: "87b0b1ee43efccf5981b67d2c81c9e04"
server_key_iv: "03e72960bba1283c77581f853f5a5039"
server_key_version: "encryptionkey@0"
I20260928 23:49:43.291420 8548 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 23:49:43.293645 8563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:43.294977 8548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260928 23:49:43.295104 8548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "dbc13a0a54194dbd89fa635b387ddb4e"
format_stamp: "Formatted at 2026-09-28 23:49:43 on dist-test-slave-p5bw"
server_key: "87b0b1ee43efccf5981b67d2c81c9e04"
server_key_iv: "03e72960bba1283c77581f853f5a5039"
server_key_version: "encryptionkey@0"
I20260928 23:49:43.295231 8548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:43.341117 8548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:43.344559 8548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:43.344775 8548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:43.346248 8548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:43.346329 8548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:43.346424 8548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:43.346477 8548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:49:43 dist-test-slave-p5bw krb5kdc[8431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639383, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:43.357380 8548 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:43429
I20260928 23:49:43.357440 8678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:43429 every 8 connection(s)
I20260928 23:49:43.358301 8548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:49:43.360138 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.347160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:53443 (local address 127.6.152.62:34125)
0928 23:49:43.347356 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:43.347363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:43.348256 (+ 893us) server_negotiation.cc:407] Connection header received
0928 23:49:43.349435 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:43.349442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:43.349521 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:43.349632 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:43.351057 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.351982 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.353544 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.353800 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.353916 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.356414 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:43.356418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:43.356420 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:43.356456 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:43.358071 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:43.358610 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:43.358614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:43.358616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:43.358723 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:43.359101 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:43.359104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:43.359106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:43.359290 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:43.359374 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:43.359901 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 23:49:43.360016 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 23:49:43.360916 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.347638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:34125 (local address 127.6.152.1:53443)
0928 23:49:43.348089 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:43.348140 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:43.349167 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:43.349844 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:43.349854 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:43.350270 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:43.350887 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:43.350906 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.352118 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.352122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:43.353420 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:43.353430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.353979 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.353984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:43.354158 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.354756 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:43.354772 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:43.356242 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:43.358186 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:43.358194 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:43.358210 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:43.358510 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:43.358832 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:43.358835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:43.358838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:43.358993 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:43.359397 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:43.359402 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:43.359627 (+ 225us) client_negotiation.cc:769] Sending connection context
0928 23:49:43.359786 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 23:49:43.360038 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":130,"threads_started":1}
I20260928 23:49:43.362160 8679 heartbeater.cc:347] Connected to a master server at 127.6.152.62:34125
I20260928 23:49:43.362289 8679 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:43.362838 8679 heartbeater.cc:511] Master 127.6.152.62:34125 requested a full tablet report, sending...
I20260928 23:49:43.364497 8475 ts_manager.cc:194] Registered new tserver with Master: dbc13a0a54194dbd89fa635b387ddb4e (127.6.152.1:43429)
I20260928 23:49:43.366086 8475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.1@KRBTEST.COM'} at 127.6.152.1:53443
I20260928 23:49:43.367920 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 8548
I20260928 23:49:43.368055 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
I20260928 23:49:43.368376 6752 external_mini_cluster.cc:1428] Setting key ad9a9bc469c5e6dfb2314df8e236b42e
I20260928 23:49:43.372300 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.364430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:46553 (local address 127.6.152.62:34125)
0928 23:49:43.364545 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:43.364548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:43.364799 (+ 251us) server_negotiation.cc:407] Connection header received
0928 23:49:43.364940 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:43.364942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:43.364979 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:43.365042 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:43.365564 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.366073 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.367232 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.367488 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.367634 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.368956 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:43.368960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:43.368963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:43.368990 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:43.370570 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:43.371036 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:43.371041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:43.371043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:43.371130 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:43.371439 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:43.371444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:43.371446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:43.371627 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:43.371741 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:43.372021 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 23:49:43.372142 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:49:43.372314 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.364383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:34125 (local address 127.6.152.1:46553)
0928 23:49:43.364712 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:43.364728 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:43.364855 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:43.365089 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:43.365092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:43.365305 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:43.365457 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:43.365465 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.366206 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.366210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:43.367102 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:43.367111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.367639 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.367665 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:43.367770 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.368365 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:43.368377 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:43.368779 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:43.370708 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:43.370712 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:43.370714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:43.370926 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:43.371237 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:43.371240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:43.371243 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:43.371340 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:43.371731 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:43.371735 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:43.371844 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 23:49:43.371962 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 23:49:43.372106 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for kudu/127.6.152.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
I20260928 23:49:43.394124 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:34125
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:43.509655 8688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:43.509900 8688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:43.509975 8688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:43.514007 8688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:43.514065 8688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:43.514108 8688 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 23:49:43.514148 8688 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:49:43.514210 8688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:49:43.519332 8688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.152.62:34125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:43.520306 8688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:43.521306 8688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:43.530717 8694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:43.530759 8693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:43.530717 8696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:43.531002 8688 server_base.cc:1034] running on GCE node
I20260928 23:49:43.531390 8688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:43.532049 8688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:43.533202 8688 hybrid_clock.cc:648] HybridClock initialized: now 1790639383533189 us; error 27 us; skew 500 ppm
Sep 28 23:49:43 dist-test-slave-p5bw krb5kdc[8431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639383, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:43.535889 8688 init.cc:377] Logged in from keytab as kudu/127.6.152.2@KRBTEST.COM (short username kudu)
I20260928 23:49:43.536741 8688 webserver.cc:460] Webserver started at http://127.6.152.2:40203/ using document root <none> and password file <none>
I20260928 23:49:43.537447 8688 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:43.537511 8688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:43.537717 8688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:43.539790 8688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "8b2255c33ad74a5dae44efa30d85c0a3"
format_stamp: "Formatted at 2026-09-28 23:49:43 on dist-test-slave-p5bw"
server_key: "d296bb3e25171b78b63dcf3817b222ab"
server_key_iv: "9ecd74bb6428efa08fda1da04e04c5e8"
server_key_version: "encryptionkey@0"
I20260928 23:49:43.540301 8688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "8b2255c33ad74a5dae44efa30d85c0a3"
format_stamp: "Formatted at 2026-09-28 23:49:43 on dist-test-slave-p5bw"
server_key: "d296bb3e25171b78b63dcf3817b222ab"
server_key_iv: "9ecd74bb6428efa08fda1da04e04c5e8"
server_key_version: "encryptionkey@0"
I20260928 23:49:43.543843 8688 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 23:49:43.546130 8703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:43.547261 8688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:49:43.547377 8688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "8b2255c33ad74a5dae44efa30d85c0a3"
format_stamp: "Formatted at 2026-09-28 23:49:43 on dist-test-slave-p5bw"
server_key: "d296bb3e25171b78b63dcf3817b222ab"
server_key_iv: "9ecd74bb6428efa08fda1da04e04c5e8"
server_key_version: "encryptionkey@0"
I20260928 23:49:43.547478 8688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:43.557264 8688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:43.560665 8688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:43.560880 8688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:43.562237 8688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:43.562320 8688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:43.562394 8688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:43.562434 8688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:49:43 dist-test-slave-p5bw krb5kdc[8431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639383, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:43.573591 8688 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:41243
I20260928 23:49:43.574417 8688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:49:43.574383 8818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:41243 every 8 connection(s)
I20260928 23:49:43.576421 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.563209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:52093 (local address 127.6.152.62:34125)
0928 23:49:43.563340 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:43.563344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:43.564375 (+ 1031us) server_negotiation.cc:407] Connection header received
0928 23:49:43.565499 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:43.565502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:43.565565 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:43.565658 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:43.567207 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.567998 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.569845 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.570101 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.570231 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.572640 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:43.572643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:43.572645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:43.572670 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:43.574191 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:43.574870 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:43.574875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:43.574877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:43.575001 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:43.575360 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:43.575363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:43.575365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:43.575556 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:43.575667 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:43.576130 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 23:49:43.576254 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:49:43.577196 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.563761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:34125 (local address 127.6.152.2:52093)
0928 23:49:43.564229 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:43.564268 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:43.565230 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:43.565853 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:43.565859 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:43.566347 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:43.567036 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:43.567061 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.568118 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.568126 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:43.569713 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:43.569727 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.570300 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.570304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:43.570466 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.571031 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:43.571047 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:43.572450 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:43.574307 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:43.574312 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:43.574327 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:43.574771 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:43.575113 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:43.575116 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:43.575118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:43.575274 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:43.575664 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:43.575669 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:43.575898 (+ 229us) client_negotiation.cc:769] Sending connection context
0928 23:49:43.576087 (+ 189us) client_negotiation.cc:240] Negotiation successful
0928 23:49:43.576380 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":136,"threads_started":1}
I20260928 23:49:43.578395 8819 heartbeater.cc:347] Connected to a master server at 127.6.152.62:34125
I20260928 23:49:43.578555 8819 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:43.579061 8819 heartbeater.cc:511] Master 127.6.152.62:34125 requested a full tablet report, sending...
I20260928 23:49:43.580077 8476 ts_manager.cc:194] Registered new tserver with Master: 8b2255c33ad74a5dae44efa30d85c0a3 (127.6.152.2:41243)
I20260928 23:49:43.580550 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 8688
I20260928 23:49:43.580667 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:49:43.580876 8476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.2@KRBTEST.COM'} at 127.6.152.2:52093
I20260928 23:49:43.581027 6752 external_mini_cluster.cc:1428] Setting key f8bc91140f3d31529c17e5123d980881
I20260928 23:49:43.588400 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.580540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:60591 (local address 127.6.152.62:34125)
0928 23:49:43.580639 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:43.580642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:43.581148 (+ 506us) server_negotiation.cc:407] Connection header received
0928 23:49:43.581294 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:43.581296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:43.581343 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:43.581424 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:43.582252 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.582726 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.583560 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.583774 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.583882 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.584945 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:43.584948 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:43.584951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:43.584985 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:43.586403 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:43.586835 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:43.586844 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:43.586846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:43.586915 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:43.587572 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:43.587575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:43.587577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:43.587780 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:43.587894 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:43.588145 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 23:49:43.588246 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 23:49:43.588407 8825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.580467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:34125 (local address 127.6.152.2:60591)
0928 23:49:43.581065 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:43.581081 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:43.581213 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:43.581607 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:43.581610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:43.581814 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:43.581917 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:43.581936 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.582870 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.582874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:43.583431 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:43.583439 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.583884 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.583887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:43.583978 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.584468 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:43.584491 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:43.584785 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:43.586519 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:43.586525 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:43.586527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:43.586728 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:43.587026 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:43.587029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:43.587031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:43.587118 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:43.587882 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:43.587886 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:43.587977 (+ 91us) client_negotiation.cc:769] Sending connection context
0928 23:49:43.588091 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 23:49:43.588231 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.6.152.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
I20260928 23:49:43.607177 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:34125
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:43.717046 8828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:43.717291 8828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:43.717337 8828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:43.721251 8828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:43.721300 8828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:43.721339 8828 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 23:49:43.721354 8828 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:49:43.721385 8828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:49:43.726532 8828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.152.62:34125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:43.727540 8828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:43.728482 8828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:43.737685 8836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:43.737685 8833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:43.737846 8834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:43.738010 8828 server_base.cc:1034] running on GCE node
I20260928 23:49:43.738623 8828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:43.739233 8828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:43.740383 8828 hybrid_clock.cc:648] HybridClock initialized: now 1790639383740364 us; error 30 us; skew 500 ppm
Sep 28 23:49:43 dist-test-slave-p5bw krb5kdc[8431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639383, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:43.743007 8828 init.cc:377] Logged in from keytab as kudu/127.6.152.3@KRBTEST.COM (short username kudu)
I20260928 23:49:43.743861 8828 webserver.cc:460] Webserver started at http://127.6.152.3:42189/ using document root <none> and password file <none>
I20260928 23:49:43.744398 8828 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:43.744489 8828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:43.744715 8828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:43.746665 8828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "ab54842e038346e2b182f5274c66d964"
format_stamp: "Formatted at 2026-09-28 23:49:43 on dist-test-slave-p5bw"
server_key: "9bda3d73ced84149b72fd003babfa500"
server_key_iv: "a8e0ee4a3dbbaa2d429681d07f482006"
server_key_version: "encryptionkey@0"
I20260928 23:49:43.747213 8828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "ab54842e038346e2b182f5274c66d964"
format_stamp: "Formatted at 2026-09-28 23:49:43 on dist-test-slave-p5bw"
server_key: "9bda3d73ced84149b72fd003babfa500"
server_key_iv: "a8e0ee4a3dbbaa2d429681d07f482006"
server_key_version: "encryptionkey@0"
I20260928 23:49:43.750831 8828 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 23:49:43.753227 8843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:43.754379 8828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:49:43.754516 8828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "ab54842e038346e2b182f5274c66d964"
format_stamp: "Formatted at 2026-09-28 23:49:43 on dist-test-slave-p5bw"
server_key: "9bda3d73ced84149b72fd003babfa500"
server_key_iv: "a8e0ee4a3dbbaa2d429681d07f482006"
server_key_version: "encryptionkey@0"
I20260928 23:49:43.754624 8828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:43.763692 8828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:43.766387 8828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:43.766587 8828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:43.767944 8828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:43.768015 8828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:43.768087 8828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:43.768141 8828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:49:43 dist-test-slave-p5bw krb5kdc[8431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639383, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:43.778592 8828 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:33245
I20260928 23:49:43.778684 8958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:33245 every 8 connection(s)
I20260928 23:49:43.779623 8828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:49:43.781925 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.768657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:55187 (local address 127.6.152.62:34125)
0928 23:49:43.768833 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:43.768839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:43.770130 (+ 1291us) server_negotiation.cc:407] Connection header received
0928 23:49:43.771559 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:43.771562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:43.771655 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:43.771752 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:43.773183 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.774095 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.775457 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.775802 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.775940 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.778479 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:43.778483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:43.778486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:43.778516 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:43.780081 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:43.780627 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:43.780630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:43.780631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:43.780707 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:43.781040 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:43.781043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:43.781044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:43.781206 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:43.781278 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:43.781716 (+ 438us) server_negotiation.cc:299] Negotiation successful
0928 23:49:43.781808 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 23:49:43.782673 8852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.769498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:34125 (local address 127.6.152.3:55187)
0928 23:49:43.769963 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:43.770028 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:43.771311 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:43.771922 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:43.771928 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:43.772437 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:43.773027 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:43.773046 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.774241 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.774245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:43.775350 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:43.775364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.775954 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.775956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:43.776088 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.776679 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:43.776704 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:43.778242 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:43.780211 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:43.780216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:43.780229 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:43.780535 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:43.780767 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:43.780769 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:43.780770 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:43.780947 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:43.781293 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:43.781297 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:43.781513 (+ 216us) client_negotiation.cc:769] Sending connection context
0928 23:49:43.781654 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 23:49:43.781882 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":102,"threads_started":1}
I20260928 23:49:43.783315 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 8828
I20260928 23:49:43.783450 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:49:43.783756 6752 external_mini_cluster.cc:1428] Setting key b1f01759e4f26b639d05fa2990958f2a
I20260928 23:49:43.783922 8959 heartbeater.cc:347] Connected to a master server at 127.6.152.62:34125
I20260928 23:49:43.784050 8959 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:43.784639 8959 heartbeater.cc:511] Master 127.6.152.62:34125 requested a full tablet report, sending...
I20260928 23:49:43.785691 8476 ts_manager.cc:194] Registered new tserver with Master: ab54842e038346e2b182f5274c66d964 (127.6.152.3:33245)
I20260928 23:49:43.786432 8476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.3@KRBTEST.COM'} at 127.6.152.3:55187
I20260928 23:49:43.793017 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.786191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:44005 (local address 127.6.152.62:34125)
0928 23:49:43.786317 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:43.786321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:43.786509 (+ 188us) server_negotiation.cc:407] Connection header received
0928 23:49:43.786608 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:43.786610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:43.786669 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:43.786738 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:43.787284 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.787781 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.788582 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.788869 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.788985 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.790025 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:43.790029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:43.790031 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:43.790054 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:43.791431 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:43.791863 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:43.791867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:43.791868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:43.791944 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:43.792225 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:43.792228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:43.792229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:43.792390 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:43.792494 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:43.792752 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 23:49:43.792881 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:49:43.793036 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.786137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:34125 (local address 127.6.152.3:44005)
0928 23:49:43.786421 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:43.786435 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:43.786530 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:43.786844 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:43.786847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:43.787065 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:43.787169 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:43.787174 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.787903 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.787906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:43.788466 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:43.788474 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.788984 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.788988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:43.789081 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.789577 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:43.789591 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:43.789884 (+ 293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:43.791563 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:43.791567 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:43.791569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:43.791757 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:43.792038 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:43.792041 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:43.792043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:43.792126 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:43.792484 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:43.792487 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:43.792576 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 23:49:43.792688 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 23:49:43.792820 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":122,"threads_started":1}
I20260928 23:49:43.797355 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 23:49:43 dist-test-slave-p5bw krb5kdc[8431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260928 23:49:43.813880 8521 acceptor_pool.cc:285] unable to collect diagnostics on RPC socket 127.6.152.62:34125: Invalid argument: socket address is ambiguous
Sep 28 23:49:43 dist-test-slave-p5bw krb5kdc[8431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:43.821636 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.814111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47674 (local address 127.6.152.62:34125)
0928 23:49:43.814240 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:43.814243 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:43.814256 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:49:43.814293 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:43.814295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:43.814330 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:43.814420 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:43.815092 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.815607 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.816460 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.816678 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.816785 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.819016 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:43.819019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:43.819021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:43.819043 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:43.820204 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:43.820570 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:43.820572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:43.820573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:43.820643 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:43.820909 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:43.820911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:43.820912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:43.821067 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:43.821164 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:43.821353 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 23:49:43.821438 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:49:43.824421 8476 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:49:43.827391 8476 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:49:43.841514 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.836136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:41243 (local address 127.0.0.1:38136)
0928 23:49:43.836465 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:43.836479 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:43.836566 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:43.837232 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:43.837238 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:43.837261 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:49:43.837407 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:43.837414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.838839 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.838842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:43.840142 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:43.840152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.840977 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.840981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:43.841093 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.841130 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 23:49:43.841238 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 23:49:43.841348 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":144,"thread_start_us":81,"threads_started":1}
I20260928 23:49:43.841656 8976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.835401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:33245 (local address 127.0.0.1:44864)
0928 23:49:43.835737 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:43.835790 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:43.835928 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:43.837029 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:43.837035 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:43.837057 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:49:43.837207 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:43.837214 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.838792 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.838795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:43.840169 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:43.840179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.841260 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.841263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:43.841341 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.841359 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 23:49:43.841428 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 23:49:43.841482 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":68,"thread_start_us":84,"threads_started":1}
I20260928 23:49:43.841895 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.835565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44864 (local address 127.6.152.3:33245)
0928 23:49:43.836503 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:43.836515 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:43.836537 (+ 22us) server_negotiation.cc:407] Connection header received
0928 23:49:43.836613 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:43.836618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:43.836852 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:43.837338 (+ 486us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:49:43.837491 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.838671 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.840322 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.841154 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.841263 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.841444 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:43.841500 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 23:49:43.841751 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":100,"threads_started":1}
I20260928 23:49:43.842145 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.836314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38136 (local address 127.6.152.2:41243)
0928 23:49:43.836719 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:43.836730 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:43.836748 (+ 18us) server_negotiation.cc:407] Connection header received
0928 23:49:43.836859 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:43.836865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:43.837092 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:43.837242 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:49:43.837526 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.838679 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.840261 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.840860 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.840981 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.841153 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:43.841893 (+ 740us) server_negotiation.cc:299] Negotiation successful
0928 23:49:43.842011 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":113,"threads_started":1}
I20260928 23:49:43.842166 8978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.835758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:43429 (local address 127.0.0.1:60812)
0928 23:49:43.836151 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:43.836163 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:43.836240 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:43.837365 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:43.837370 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:43.837387 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:49:43.837480 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:43.837488 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.838670 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.838677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:43.840365 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:43.840373 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.841524 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.841528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:43.841844 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.841869 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 23:49:43.841953 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 23:49:43.842035 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":173,"thread_start_us":99,"threads_started":1}
I20260928 23:49:43.842849 8979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.836070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60812 (local address 127.6.152.1:43429)
0928 23:49:43.836793 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:43.836807 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:43.836854 (+ 47us) server_negotiation.cc:407] Connection header received
0928 23:49:43.836932 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:43.836937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:43.837141 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:43.837613 (+ 472us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:49:43.837711 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.838571 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.840484 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.841246 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.841334 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.841464 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:43.841998 (+ 534us) server_negotiation.cc:299] Negotiation successful
0928 23:49:43.842085 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":84,"threads_started":1}
I20260928 23:49:43.844614 8753 tablet_service.cc:1467] Processing CreateTablet for tablet 60c1903310c44123b028bd9f6b9af356 (DEFAULT_TABLE table=test-table [id=225a10717885469a9f18c69c39ce2321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:43.844749 8893 tablet_service.cc:1467] Processing CreateTablet for tablet 60c1903310c44123b028bd9f6b9af356 (DEFAULT_TABLE table=test-table [id=225a10717885469a9f18c69c39ce2321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:43.845122 8613 tablet_service.cc:1467] Processing CreateTablet for tablet 60c1903310c44123b028bd9f6b9af356 (DEFAULT_TABLE table=test-table [id=225a10717885469a9f18c69c39ce2321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:43.845724 8753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c1903310c44123b028bd9f6b9af356. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:43.845743 8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c1903310c44123b028bd9f6b9af356. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:43.845989 8613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c1903310c44123b028bd9f6b9af356. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:43.852268 8982 tablet_bootstrap.cc:492] T 60c1903310c44123b028bd9f6b9af356 P 8b2255c33ad74a5dae44efa30d85c0a3: Bootstrap starting.
I20260928 23:49:43.853786 8983 tablet_bootstrap.cc:492] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964: Bootstrap starting.
I20260928 23:49:43.854667 8982 tablet_bootstrap.cc:654] T 60c1903310c44123b028bd9f6b9af356 P 8b2255c33ad74a5dae44efa30d85c0a3: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:43.855361 8982 log.cc:824] T 60c1903310c44123b028bd9f6b9af356 P 8b2255c33ad74a5dae44efa30d85c0a3: Log is configured to *not* fsync() on all Append() calls
I20260928 23:49:43.855607 8983 tablet_bootstrap.cc:654] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:43.856040 8984 tablet_bootstrap.cc:492] T 60c1903310c44123b028bd9f6b9af356 P dbc13a0a54194dbd89fa635b387ddb4e: Bootstrap starting.
I20260928 23:49:43.856344 8983 log.cc:824] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964: Log is configured to *not* fsync() on all Append() calls
I20260928 23:49:43.857390 8982 tablet_bootstrap.cc:492] T 60c1903310c44123b028bd9f6b9af356 P 8b2255c33ad74a5dae44efa30d85c0a3: No bootstrap required, opened a new log
I20260928 23:49:43.857573 8982 ts_tablet_manager.cc:1397] T 60c1903310c44123b028bd9f6b9af356 P 8b2255c33ad74a5dae44efa30d85c0a3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 23:49:43.858004 8984 tablet_bootstrap.cc:654] T 60c1903310c44123b028bd9f6b9af356 P dbc13a0a54194dbd89fa635b387ddb4e: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:43.858258 8983 tablet_bootstrap.cc:492] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964: No bootstrap required, opened a new log
I20260928 23:49:43.858443 8983 ts_tablet_manager.cc:1397] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 23:49:43.858768 8984 log.cc:824] T 60c1903310c44123b028bd9f6b9af356 P dbc13a0a54194dbd89fa635b387ddb4e: Log is configured to *not* fsync() on all Append() calls
I20260928 23:49:43.860280 8984 tablet_bootstrap.cc:492] T 60c1903310c44123b028bd9f6b9af356 P dbc13a0a54194dbd89fa635b387ddb4e: No bootstrap required, opened a new log
I20260928 23:49:43.860489 8984 ts_tablet_manager.cc:1397] T 60c1903310c44123b028bd9f6b9af356 P dbc13a0a54194dbd89fa635b387ddb4e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 23:49:43.862164 8982 raft_consensus.cc:348] T 60c1903310c44123b028bd9f6b9af356 P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } }
I20260928 23:49:43.862164 8983 raft_consensus.cc:348] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } }
I20260928 23:49:43.862440 8982 raft_consensus.cc:374] T 60c1903310c44123b028bd9f6b9af356 P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:43.862440 8983 raft_consensus.cc:374] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:43.862499 8983 raft_consensus.cc:729] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab54842e038346e2b182f5274c66d964, State: Initialized, Role: FOLLOWER
I20260928 23:49:43.862499 8982 raft_consensus.cc:729] T 60c1903310c44123b028bd9f6b9af356 P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b2255c33ad74a5dae44efa30d85c0a3, State: Initialized, Role: FOLLOWER
I20260928 23:49:43.862951 8982 consensus_queue.cc:260] T 60c1903310c44123b028bd9f6b9af356 P 8b2255c33ad74a5dae44efa30d85c0a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } }
I20260928 23:49:43.862951 8983 consensus_queue.cc:260] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } }
I20260928 23:49:43.863654 8819 heartbeater.cc:503] Master 127.6.152.62:34125 was elected leader, sending a full tablet report...
I20260928 23:49:43.863899 8959 heartbeater.cc:503] Master 127.6.152.62:34125 was elected leader, sending a full tablet report...
I20260928 23:49:43.863917 8982 ts_tablet_manager.cc:1428] T 60c1903310c44123b028bd9f6b9af356 P 8b2255c33ad74a5dae44efa30d85c0a3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:49:43.863746 8984 raft_consensus.cc:348] T 60c1903310c44123b028bd9f6b9af356 P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } }
I20260928 23:49:43.863981 8984 raft_consensus.cc:374] T 60c1903310c44123b028bd9f6b9af356 P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:43.864027 8984 raft_consensus.cc:729] T 60c1903310c44123b028bd9f6b9af356 P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbc13a0a54194dbd89fa635b387ddb4e, State: Initialized, Role: FOLLOWER
I20260928 23:49:43.864420 8984 consensus_queue.cc:260] T 60c1903310c44123b028bd9f6b9af356 P dbc13a0a54194dbd89fa635b387ddb4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } }
I20260928 23:49:43.864945 8983 ts_tablet_manager.cc:1428] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 23:49:43.865425 8679 heartbeater.cc:503] Master 127.6.152.62:34125 was elected leader, sending a full tablet report...
I20260928 23:49:43.865700 8984 ts_tablet_manager.cc:1428] T 60c1903310c44123b028bd9f6b9af356 P dbc13a0a54194dbd89fa635b387ddb4e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 23:49:43.921269 8989 raft_consensus.cc:482] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:49:43.921458 8989 raft_consensus.cc:504] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } }
I20260928 23:49:43.922541 8989 leader_election.cc:288] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b2255c33ad74a5dae44efa30d85c0a3 (127.6.152.2:41243), dbc13a0a54194dbd89fa635b387ddb4e (127.6.152.1:43429)
I20260928 23:49:43.926193 8852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.922769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:43429 (local address 127.6.152.3:47285)
0928 23:49:43.922932 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:43.922948 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:43.923041 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:43.923340 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:43.923343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:43.923360 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:49:43.923481 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:43.923487 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.924225 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.924231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:43.925082 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:43.925091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.925742 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.925746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:43.925879 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.925897 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 23:49:43.925987 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 23:49:43.926064 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":1}
I20260928 23:49:43.926193 8979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.922900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:47285 (local address 127.6.152.1:43429)
0928 23:49:43.923030 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:43.923036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:43.923053 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:49:43.923186 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:43.923189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:43.923233 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:43.923327 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:49:43.923603 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.924109 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.925207 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.925622 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.925743 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.925814 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:43.926002 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 23:49:43.926074 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:49:43.926863 8633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c1903310c44123b028bd9f6b9af356" candidate_uuid: "ab54842e038346e2b182f5274c66d964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" is_pre_election: true
I20260928 23:49:43.926975 8991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.922771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:41243 (local address 127.6.152.3:43977)
0928 23:49:43.923379 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:43.923390 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:43.923503 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:43.923808 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:43.923811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:43.923827 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:49:43.923957 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:43.923962 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.924660 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.924666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:43.925805 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:43.925815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.926573 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.926577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:43.926673 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.926696 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:49:43.926771 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 23:49:43.926832 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":267,"thread_start_us":159,"threads_started":1}
I20260928 23:49:43.927044 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.922958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:43977 (local address 127.6.152.2:41243)
0928 23:49:43.923087 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:43.923092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:43.923472 (+ 380us) server_negotiation.cc:407] Connection header received
0928 23:49:43.923666 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:43.923668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:43.923710 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:43.923787 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:49:43.924064 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.924555 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.925927 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.926435 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.926555 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.926619 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:43.926790 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 23:49:43.926879 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 23:49:43.927206 8633 raft_consensus.cc:2454] T 60c1903310c44123b028bd9f6b9af356 P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab54842e038346e2b182f5274c66d964 in term 0.
I20260928 23:49:43.927626 8773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c1903310c44123b028bd9f6b9af356" candidate_uuid: "ab54842e038346e2b182f5274c66d964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" is_pre_election: true
I20260928 23:49:43.927721 8846 leader_election.cc:302] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab54842e038346e2b182f5274c66d964, dbc13a0a54194dbd89fa635b387ddb4e; no voters:
I20260928 23:49:43.927901 8773 raft_consensus.cc:2454] T 60c1903310c44123b028bd9f6b9af356 P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab54842e038346e2b182f5274c66d964 in term 0.
I20260928 23:49:43.928061 8989 raft_consensus.cc:2781] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:49:43.928165 8989 raft_consensus.cc:482] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:49:43.928222 8989 raft_consensus.cc:3037] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:43.929327 8989 raft_consensus.cc:504] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } }
I20260928 23:49:43.929697 8989 leader_election.cc:288] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964 [CANDIDATE]: Term 1 election: Requested vote from peers 8b2255c33ad74a5dae44efa30d85c0a3 (127.6.152.2:41243), dbc13a0a54194dbd89fa635b387ddb4e (127.6.152.1:43429)
I20260928 23:49:43.930027 8773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c1903310c44123b028bd9f6b9af356" candidate_uuid: "ab54842e038346e2b182f5274c66d964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b2255c33ad74a5dae44efa30d85c0a3"
I20260928 23:49:43.930066 8633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c1903310c44123b028bd9f6b9af356" candidate_uuid: "ab54842e038346e2b182f5274c66d964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc13a0a54194dbd89fa635b387ddb4e"
I20260928 23:49:43.930128 8773 raft_consensus.cc:3037] T 60c1903310c44123b028bd9f6b9af356 P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:43.930235 8633 raft_consensus.cc:3037] T 60c1903310c44123b028bd9f6b9af356 P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:43.931082 8773 raft_consensus.cc:2454] T 60c1903310c44123b028bd9f6b9af356 P 8b2255c33ad74a5dae44efa30d85c0a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab54842e038346e2b182f5274c66d964 in term 1.
I20260928 23:49:43.931274 8633 raft_consensus.cc:2454] T 60c1903310c44123b028bd9f6b9af356 P dbc13a0a54194dbd89fa635b387ddb4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab54842e038346e2b182f5274c66d964 in term 1.
I20260928 23:49:43.931367 8847 leader_election.cc:302] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b2255c33ad74a5dae44efa30d85c0a3, ab54842e038346e2b182f5274c66d964; no voters:
I20260928 23:49:43.931566 8989 raft_consensus.cc:2781] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:49:43.931795 8989 raft_consensus.cc:686] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964 [term 1 LEADER]: Becoming Leader. State: Replica: ab54842e038346e2b182f5274c66d964, State: Running, Role: LEADER
I20260928 23:49:43.932135 8989 consensus_queue.cc:237] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } }
I20260928 23:49:43.935292 8476 catalog_manager.cc:5495] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab54842e038346e2b182f5274c66d964 (127.6.152.3). New cstate: current_term: 1 leader_uuid: "ab54842e038346e2b182f5274c66d964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } health_report { overall_health: UNKNOWN } } }
Sep 28 23:49:43 dist-test-slave-p5bw krb5kdc[8431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639383, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:43.991012 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.982612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:34125 (local address 127.0.0.1:47680)
0928 23:49:43.982945 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:43.982957 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:43.983103 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:43.983388 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:43.983390 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:43.983814 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:43.983903 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:43.983908 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.984692 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.984698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:43.985275 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:43.985283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.985743 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.985746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:43.985825 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.986412 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:43.986424 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:43.987770 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:43.989451 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:43.989461 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:43.989468 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:43.989701 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:43.989977 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:43.989980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:43.989982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:43.990061 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:43.990423 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:43.990430 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:43.990546 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 23:49:43.990660 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 23:49:43.990808 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":96,"threads_started":1}
I20260928 23:49:43.991016 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:43.982707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47680 (local address 127.6.152.62:34125)
0928 23:49:43.982854 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:43.982859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:43.983047 (+ 188us) server_negotiation.cc:407] Connection header received
0928 23:49:43.983212 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:43.983215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:43.983258 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:43.983339 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:43.984029 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.984588 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.985392 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:43.985606 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:43.985740 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:43.987898 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:43.987902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:43.987904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:43.987933 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:43.989326 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:43.989797 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:43.989803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:43.989804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:43.989879 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:43.990148 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:43.990153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:43.990155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:43.990320 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:43.990423 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:43.990715 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 23:49:43.990859 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":50}
I20260928 23:49:43.995044 8476 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:47680:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 23:49:43.995800 8476 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:49:44.000720 8893 tablet_service.cc:1467] Processing CreateTablet for tablet f56e81da1b3445339bc5ca6dc37e5269 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5379e52e507f40699325ec81d35083cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:49:44.000667 8613 tablet_service.cc:1467] Processing CreateTablet for tablet f56e81da1b3445339bc5ca6dc37e5269 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5379e52e507f40699325ec81d35083cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:49:44.000941 8753 tablet_service.cc:1467] Processing CreateTablet for tablet f56e81da1b3445339bc5ca6dc37e5269 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5379e52e507f40699325ec81d35083cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:49:44.001428 8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f56e81da1b3445339bc5ca6dc37e5269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:44.001428 8613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f56e81da1b3445339bc5ca6dc37e5269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:44.001641 8753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f56e81da1b3445339bc5ca6dc37e5269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:44.004225 8982 tablet_bootstrap.cc:492] T f56e81da1b3445339bc5ca6dc37e5269 P 8b2255c33ad74a5dae44efa30d85c0a3: Bootstrap starting.
I20260928 23:49:44.004241 8984 tablet_bootstrap.cc:492] T f56e81da1b3445339bc5ca6dc37e5269 P dbc13a0a54194dbd89fa635b387ddb4e: Bootstrap starting.
I20260928 23:49:44.004241 8983 tablet_bootstrap.cc:492] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964: Bootstrap starting.
I20260928 23:49:44.005134 8982 tablet_bootstrap.cc:654] T f56e81da1b3445339bc5ca6dc37e5269 P 8b2255c33ad74a5dae44efa30d85c0a3: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:44.005195 8984 tablet_bootstrap.cc:654] T f56e81da1b3445339bc5ca6dc37e5269 P dbc13a0a54194dbd89fa635b387ddb4e: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:44.005195 8983 tablet_bootstrap.cc:654] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:44.006503 8983 tablet_bootstrap.cc:492] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964: No bootstrap required, opened a new log
I20260928 23:49:44.006563 8984 tablet_bootstrap.cc:492] T f56e81da1b3445339bc5ca6dc37e5269 P dbc13a0a54194dbd89fa635b387ddb4e: No bootstrap required, opened a new log
I20260928 23:49:44.006579 8983 ts_tablet_manager.cc:1397] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 23:49:44.006656 8984 ts_tablet_manager.cc:1397] T f56e81da1b3445339bc5ca6dc37e5269 P dbc13a0a54194dbd89fa635b387ddb4e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 23:49:44.007000 8983 raft_consensus.cc:348] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:44.007105 8983 raft_consensus.cc:374] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:44.007032 8984 raft_consensus.cc:348] T f56e81da1b3445339bc5ca6dc37e5269 P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:44.007128 8983 raft_consensus.cc:729] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab54842e038346e2b182f5274c66d964, State: Initialized, Role: FOLLOWER
I20260928 23:49:44.007143 8984 raft_consensus.cc:374] T f56e81da1b3445339bc5ca6dc37e5269 P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:44.007184 8984 raft_consensus.cc:729] T f56e81da1b3445339bc5ca6dc37e5269 P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbc13a0a54194dbd89fa635b387ddb4e, State: Initialized, Role: FOLLOWER
I20260928 23:49:44.007275 8983 consensus_queue.cc:260] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:44.007311 8984 consensus_queue.cc:260] T f56e81da1b3445339bc5ca6dc37e5269 P dbc13a0a54194dbd89fa635b387ddb4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:44.007570 8983 ts_tablet_manager.cc:1428] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:49:44.007603 8984 ts_tablet_manager.cc:1428] T f56e81da1b3445339bc5ca6dc37e5269 P dbc13a0a54194dbd89fa635b387ddb4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 23:49:44.007568 8990 tablet_replica.cc:439] T f56e81da1b3445339bc5ca6dc37e5269 P dbc13a0a54194dbd89fa635b387ddb4e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } } }
I20260928 23:49:44.007705 8990 tablet_replica.cc:442] T f56e81da1b3445339bc5ca6dc37e5269 P dbc13a0a54194dbd89fa635b387ddb4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:49:44.008011 8989 tablet_replica.cc:439] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } } }
I20260928 23:49:44.008131 8989 tablet_replica.cc:442] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:49:44.008352 8982 tablet_bootstrap.cc:492] T f56e81da1b3445339bc5ca6dc37e5269 P 8b2255c33ad74a5dae44efa30d85c0a3: No bootstrap required, opened a new log
I20260928 23:49:44.008432 8982 ts_tablet_manager.cc:1397] T f56e81da1b3445339bc5ca6dc37e5269 P 8b2255c33ad74a5dae44efa30d85c0a3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 23:49:44.008909 8982 raft_consensus.cc:348] T f56e81da1b3445339bc5ca6dc37e5269 P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:44.009044 8982 raft_consensus.cc:374] T f56e81da1b3445339bc5ca6dc37e5269 P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:44.009088 8982 raft_consensus.cc:729] T f56e81da1b3445339bc5ca6dc37e5269 P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b2255c33ad74a5dae44efa30d85c0a3, State: Initialized, Role: FOLLOWER
I20260928 23:49:44.009225 8982 consensus_queue.cc:260] T f56e81da1b3445339bc5ca6dc37e5269 P 8b2255c33ad74a5dae44efa30d85c0a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:44.009534 8982 ts_tablet_manager.cc:1428] T f56e81da1b3445339bc5ca6dc37e5269 P 8b2255c33ad74a5dae44efa30d85c0a3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:49:44.009572 8988 tablet_replica.cc:439] T f56e81da1b3445339bc5ca6dc37e5269 P 8b2255c33ad74a5dae44efa30d85c0a3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } } }
I20260928 23:49:44.009717 8988 tablet_replica.cc:442] T f56e81da1b3445339bc5ca6dc37e5269 P 8b2255c33ad74a5dae44efa30d85c0a3: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 23:49:44.030690 8960 tablet.cc:2369] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:49:44.073807 8989 raft_consensus.cc:482] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:49:44.073953 8989 raft_consensus.cc:504] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:44.074397 8989 leader_election.cc:288] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbc13a0a54194dbd89fa635b387ddb4e (127.6.152.1:43429), 8b2255c33ad74a5dae44efa30d85c0a3 (127.6.152.2:41243)
I20260928 23:49:44.074807 8773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56e81da1b3445339bc5ca6dc37e5269" candidate_uuid: "ab54842e038346e2b182f5274c66d964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" is_pre_election: true
I20260928 23:49:44.074820 8633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56e81da1b3445339bc5ca6dc37e5269" candidate_uuid: "ab54842e038346e2b182f5274c66d964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" is_pre_election: true
I20260928 23:49:44.074957 8773 raft_consensus.cc:2454] T f56e81da1b3445339bc5ca6dc37e5269 P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab54842e038346e2b182f5274c66d964 in term 0.
I20260928 23:49:44.074980 8633 raft_consensus.cc:2454] T f56e81da1b3445339bc5ca6dc37e5269 P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab54842e038346e2b182f5274c66d964 in term 0.
I20260928 23:49:44.075279 8846 leader_election.cc:302] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8b2255c33ad74a5dae44efa30d85c0a3, ab54842e038346e2b182f5274c66d964, dbc13a0a54194dbd89fa635b387ddb4e; no voters:
I20260928 23:49:44.075421 8989 raft_consensus.cc:2781] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:49:44.075497 8989 raft_consensus.cc:482] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:49:44.075546 8989 raft_consensus.cc:3037] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Advancing to term 1
W20260928 23:49:44.075672 8820 tablet.cc:2369] T 60c1903310c44123b028bd9f6b9af356 P 8b2255c33ad74a5dae44efa30d85c0a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:49:44.076789 8517 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.6.152.62:34125
I20260928 23:49:44.076843 8989 raft_consensus.cc:504] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:44.077185 8989 leader_election.cc:288] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964 [CANDIDATE]: Term 1 election: Requested vote from peers dbc13a0a54194dbd89fa635b387ddb4e (127.6.152.1:43429), 8b2255c33ad74a5dae44efa30d85c0a3 (127.6.152.2:41243)
I20260928 23:49:44.077886 8773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56e81da1b3445339bc5ca6dc37e5269" candidate_uuid: "ab54842e038346e2b182f5274c66d964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b2255c33ad74a5dae44efa30d85c0a3"
I20260928 23:49:44.078060 8773 raft_consensus.cc:3037] T f56e81da1b3445339bc5ca6dc37e5269 P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:44.078199 8633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56e81da1b3445339bc5ca6dc37e5269" candidate_uuid: "ab54842e038346e2b182f5274c66d964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc13a0a54194dbd89fa635b387ddb4e"
I20260928 23:49:44.078339 8633 raft_consensus.cc:3037] T f56e81da1b3445339bc5ca6dc37e5269 P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:44.079344 8773 raft_consensus.cc:2454] T f56e81da1b3445339bc5ca6dc37e5269 P 8b2255c33ad74a5dae44efa30d85c0a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab54842e038346e2b182f5274c66d964 in term 1.
I20260928 23:49:44.079615 8847 leader_election.cc:302] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b2255c33ad74a5dae44efa30d85c0a3, ab54842e038346e2b182f5274c66d964; no voters:
I20260928 23:49:44.079784 8989 raft_consensus.cc:2781] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:49:44.079875 8633 raft_consensus.cc:2454] T f56e81da1b3445339bc5ca6dc37e5269 P dbc13a0a54194dbd89fa635b387ddb4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab54842e038346e2b182f5274c66d964 in term 1.
I20260928 23:49:44.080034 8989 raft_consensus.cc:686] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964 [term 1 LEADER]: Becoming Leader. State: Replica: ab54842e038346e2b182f5274c66d964, State: Running, Role: LEADER
I20260928 23:49:44.080173 8989 consensus_queue.cc:237] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:44.081212 8992 tablet_replica.cc:439] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964: TxnStatusTablet state changed. Reason: New leader ab54842e038346e2b182f5274c66d964. Latest consensus state: current_term: 1 leader_uuid: "ab54842e038346e2b182f5274c66d964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } } }
I20260928 23:49:44.081317 8992 tablet_replica.cc:442] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:49:44.081633 9004 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 23:49:44.081875 9004 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 23:49:44.082224 8474 catalog_manager.cc:5495] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab54842e038346e2b182f5274c66d964 (127.6.152.3). New cstate: current_term: 1 leader_uuid: "ab54842e038346e2b182f5274c66d964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } health_report { overall_health: HEALTHY } } }
I20260928 23:49:44.108067 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:44.104053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:33245 (local address 127.0.0.1:44868)
0928 23:49:44.104220 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:44.104242 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:44.104329 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:44.104658 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:44.104661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:44.104666 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:49:44.104756 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:44.104761 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:44.105757 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:44.105760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:44.106346 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:44.106353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:44.106913 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:44.106915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:44.106983 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:44.107028 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:49:44.107828 (+ 800us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:49:44.107838 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 23:49:44.107879 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 23:49:44.107945 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 23:49:44.108093 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:44.104150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44868 (local address 127.6.152.3:33245)
0928 23:49:44.104279 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:44.104284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:44.104299 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:49:44.104470 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:44.104473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:44.104538 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:44.104645 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:49:44.104888 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:44.105658 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:44.106455 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:44.106798 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:44.106907 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:44.107179 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:49:44.107740 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:49:44.107860 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:44.107893 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 23:49:44.107958 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 23:49:44.109697 8680 tablet.cc:2369] T 60c1903310c44123b028bd9f6b9af356 P dbc13a0a54194dbd89fa635b387ddb4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:49:44.113583 8633 raft_consensus.cc:1261] T f56e81da1b3445339bc5ca6dc37e5269 P dbc13a0a54194dbd89fa635b387ddb4e [term 1 FOLLOWER]: Refusing update from remote peer ab54842e038346e2b182f5274c66d964: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:49:44.113615 8773 raft_consensus.cc:1261] T f56e81da1b3445339bc5ca6dc37e5269 P 8b2255c33ad74a5dae44efa30d85c0a3 [term 1 FOLLOWER]: Refusing update from remote peer ab54842e038346e2b182f5274c66d964: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:49:44.114269 8989 consensus_queue.cc:1035] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:49:44.114526 8992 consensus_queue.cc:1035] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:49:44.117813 8988 tablet_replica.cc:439] T f56e81da1b3445339bc5ca6dc37e5269 P 8b2255c33ad74a5dae44efa30d85c0a3: TxnStatusTablet state changed. Reason: New leader ab54842e038346e2b182f5274c66d964. Latest consensus state: current_term: 1 leader_uuid: "ab54842e038346e2b182f5274c66d964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } } }
I20260928 23:49:44.117935 8988 tablet_replica.cc:442] T f56e81da1b3445339bc5ca6dc37e5269 P 8b2255c33ad74a5dae44efa30d85c0a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:49:44.118680 8990 tablet_replica.cc:439] T f56e81da1b3445339bc5ca6dc37e5269 P dbc13a0a54194dbd89fa635b387ddb4e: TxnStatusTablet state changed. Reason: New leader ab54842e038346e2b182f5274c66d964. Latest consensus state: current_term: 1 leader_uuid: "ab54842e038346e2b182f5274c66d964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } } }
I20260928 23:49:44.118784 8990 tablet_replica.cc:442] T f56e81da1b3445339bc5ca6dc37e5269 P dbc13a0a54194dbd89fa635b387ddb4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:49:44.121654 8992 tablet_replica.cc:439] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab54842e038346e2b182f5274c66d964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } } }
I20260928 23:49:44.121766 8992 tablet_replica.cc:442] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:49:44.121871 8992 tablet_replica.cc:439] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab54842e038346e2b182f5274c66d964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } } }
I20260928 23:49:44.121933 8992 tablet_replica.cc:442] T f56e81da1b3445339bc5ca6dc37e5269 P ab54842e038346e2b182f5274c66d964: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:49:44.122905 8988 tablet_replica.cc:439] T f56e81da1b3445339bc5ca6dc37e5269 P 8b2255c33ad74a5dae44efa30d85c0a3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab54842e038346e2b182f5274c66d964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } } }
I20260928 23:49:44.123005 8988 tablet_replica.cc:442] T f56e81da1b3445339bc5ca6dc37e5269 P 8b2255c33ad74a5dae44efa30d85c0a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:49:44.124663 9009 mvcc.cc:205] Tried to move back new op lower bound from 7334458917322362880 to 7334458917193883648. Current Snapshot: MvccSnapshot[applied={T|T < 7334458917322362880}]
I20260928 23:49:44.126266 8990 tablet_replica.cc:439] T f56e81da1b3445339bc5ca6dc37e5269 P dbc13a0a54194dbd89fa635b387ddb4e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab54842e038346e2b182f5274c66d964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } } }
I20260928 23:49:44.126376 8990 tablet_replica.cc:442] T f56e81da1b3445339bc5ca6dc37e5269 P dbc13a0a54194dbd89fa635b387ddb4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:49:44.132162 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:44.128833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44882 (local address 127.6.152.3:33245)
0928 23:49:44.128942 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:44.128947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:44.128966 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:49:44.129112 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:44.129115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:44.129178 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:44.129258 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:49:44.129476 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:44.130252 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:44.131080 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:44.131422 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:44.131520 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:44.131756 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:49:44.131841 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:49:44.131910 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:44.131984 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 23:49:44.132032 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 23:49:44.143476 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:44.139661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:33245 (local address 127.6.152.3:33435)
0928 23:49:44.139895 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:44.139907 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:44.139979 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:44.140175 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:44.140177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:44.140182 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:49:44.140265 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:44.140269 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:44.141230 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:44.141232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:44.142281 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:44.142289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:44.142832 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:44.142835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:44.142901 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:44.142939 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:49:44.143223 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:49:44.143227 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:49:44.143277 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 23:49:44.143335 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260928 23:49:44.143498 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:44.139767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:33435 (local address 127.6.152.3:33245)
0928 23:49:44.139886 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:44.139891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:44.139948 (+ 57us) server_negotiation.cc:407] Connection header received
0928 23:49:44.140043 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:44.140046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:44.140089 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:44.140162 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:49:44.140359 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:44.141140 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:44.142396 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:44.142723 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:44.142844 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:44.143073 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:49:44.143145 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:49:44.143217 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:44.143289 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 23:49:44.143334 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 23:49:44.149873 8633 raft_consensus.cc:1261] T 60c1903310c44123b028bd9f6b9af356 P dbc13a0a54194dbd89fa635b387ddb4e [term 1 FOLLOWER]: Refusing update from remote peer ab54842e038346e2b182f5274c66d964: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:49:44.149887 8773 raft_consensus.cc:1261] T 60c1903310c44123b028bd9f6b9af356 P 8b2255c33ad74a5dae44efa30d85c0a3 [term 1 FOLLOWER]: Refusing update from remote peer ab54842e038346e2b182f5274c66d964: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:49:44.150276 9012 consensus_queue.cc:1035] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:49:44.150466 8992 consensus_queue.cc:1035] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:49:44.155326 9007 mvcc.cc:205] Tried to move back new op lower bound from 7334458917474578432 to 7334458916589342720. Current Snapshot: MvccSnapshot[applied={T|T < 7334458917474578432}]
I20260928 23:49:44.159724 9008 mvcc.cc:205] Tried to move back new op lower bound from 7334458917474578432 to 7334458916589342720. Current Snapshot: MvccSnapshot[applied={T|T < 7334458917474578432}]
I20260928 23:49:44.187469 8474 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:49:44.187659 8474 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:49:44.189896 8474 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c: Sending DeleteTablet for 3 replicas of tablet 60c1903310c44123b028bd9f6b9af356
I20260928 23:49:44.190554 8893 tablet_service.cc:1514] Processing DeleteTablet for tablet 60c1903310c44123b028bd9f6b9af356 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:49:44 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:44864
I20260928 23:49:44.190768 8753 tablet_service.cc:1514] Processing DeleteTablet for tablet 60c1903310c44123b028bd9f6b9af356 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:49:44 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:38136
I20260928 23:49:44.191234 9024 tablet_replica.cc:330] T 60c1903310c44123b028bd9f6b9af356 P 8b2255c33ad74a5dae44efa30d85c0a3: stopping tablet replica
I20260928 23:49:44.191181 8473 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 23:49:44.191223 8613 tablet_service.cc:1514] Processing DeleteTablet for tablet 60c1903310c44123b028bd9f6b9af356 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:49:44 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:60812
W20260928 23:49:44.191598 8473 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:49:44.191653 9023 tablet_replica.cc:330] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964: stopping tablet replica
I20260928 23:49:44.191848 9024 raft_consensus.cc:2229] T 60c1903310c44123b028bd9f6b9af356 P 8b2255c33ad74a5dae44efa30d85c0a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:49:44.191929 9023 raft_consensus.cc:2229] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:49:44.192142 9024 raft_consensus.cc:2258] T 60c1903310c44123b028bd9f6b9af356 P 8b2255c33ad74a5dae44efa30d85c0a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:49:44.192288 9023 raft_consensus.cc:2258] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:49:44.193778 9023 ts_tablet_manager.cc:1905] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:49:44.193780 9024 ts_tablet_manager.cc:1905] T 60c1903310c44123b028bd9f6b9af356 P 8b2255c33ad74a5dae44efa30d85c0a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:49:44.196768 9024 ts_tablet_manager.cc:1918] T 60c1903310c44123b028bd9f6b9af356 P 8b2255c33ad74a5dae44efa30d85c0a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:49:44.196779 9023 ts_tablet_manager.cc:1918] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:49:44.196888 9023 log.cc:1196] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/60c1903310c44123b028bd9f6b9af356
I20260928 23:49:44.197007 9024 log.cc:1196] T 60c1903310c44123b028bd9f6b9af356 P 8b2255c33ad74a5dae44efa30d85c0a3: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-1/wal/wals/60c1903310c44123b028bd9f6b9af356
I20260928 23:49:44.197211 9023 ts_tablet_manager.cc:1939] T 60c1903310c44123b028bd9f6b9af356 P ab54842e038346e2b182f5274c66d964: Deleting consensus metadata
I20260928 23:49:44.197212 8613 tablet_service.cc:1467] Processing CreateTablet for tablet b530d8f9feb840e6a5d929dddf550e6d (DEFAULT_TABLE table=test-table [id=4483a8d9317747d783902a9a8bfa4708]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:44.197448 9024 ts_tablet_manager.cc:1939] T 60c1903310c44123b028bd9f6b9af356 P 8b2255c33ad74a5dae44efa30d85c0a3: Deleting consensus metadata
I20260928 23:49:44.197463 8613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b530d8f9feb840e6a5d929dddf550e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:44.198290 8463 catalog_manager.cc:4841] TS ab54842e038346e2b182f5274c66d964 (127.6.152.3:33245): tablet 60c1903310c44123b028bd9f6b9af356 (table test-table [id=225a10717885469a9f18c69c39ce2321]) successfully deleted
I20260928 23:49:44.198740 8463 catalog_manager.cc:4841] TS 8b2255c33ad74a5dae44efa30d85c0a3 (127.6.152.2:41243): tablet 60c1903310c44123b028bd9f6b9af356 (table test-table [id=225a10717885469a9f18c69c39ce2321]) successfully deleted
I20260928 23:49:44.199731 8984 tablet_bootstrap.cc:492] T b530d8f9feb840e6a5d929dddf550e6d P dbc13a0a54194dbd89fa635b387ddb4e: Bootstrap starting.
I20260928 23:49:44.199893 8753 tablet_service.cc:1467] Processing CreateTablet for tablet b530d8f9feb840e6a5d929dddf550e6d (DEFAULT_TABLE table=test-table [id=4483a8d9317747d783902a9a8bfa4708]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:44.200103 8753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b530d8f9feb840e6a5d929dddf550e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:44.200623 8984 tablet_bootstrap.cc:654] T b530d8f9feb840e6a5d929dddf550e6d P dbc13a0a54194dbd89fa635b387ddb4e: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:44.201848 8984 tablet_bootstrap.cc:492] T b530d8f9feb840e6a5d929dddf550e6d P dbc13a0a54194dbd89fa635b387ddb4e: No bootstrap required, opened a new log
I20260928 23:49:44.201877 8982 tablet_bootstrap.cc:492] T b530d8f9feb840e6a5d929dddf550e6d P 8b2255c33ad74a5dae44efa30d85c0a3: Bootstrap starting.
I20260928 23:49:44.201936 8984 ts_tablet_manager.cc:1397] T b530d8f9feb840e6a5d929dddf550e6d P dbc13a0a54194dbd89fa635b387ddb4e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 23:49:44.202055 8893 tablet_service.cc:1467] Processing CreateTablet for tablet b530d8f9feb840e6a5d929dddf550e6d (DEFAULT_TABLE table=test-table [id=4483a8d9317747d783902a9a8bfa4708]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:44.202275 8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b530d8f9feb840e6a5d929dddf550e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:44.202353 8984 raft_consensus.cc:348] T b530d8f9feb840e6a5d929dddf550e6d P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:44.202457 8984 raft_consensus.cc:374] T b530d8f9feb840e6a5d929dddf550e6d P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:44.202491 8984 raft_consensus.cc:729] T b530d8f9feb840e6a5d929dddf550e6d P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbc13a0a54194dbd89fa635b387ddb4e, State: Initialized, Role: FOLLOWER
I20260928 23:49:44.202610 8984 consensus_queue.cc:260] T b530d8f9feb840e6a5d929dddf550e6d P dbc13a0a54194dbd89fa635b387ddb4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:44.202740 8982 tablet_bootstrap.cc:654] T b530d8f9feb840e6a5d929dddf550e6d P 8b2255c33ad74a5dae44efa30d85c0a3: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:44.202893 8984 ts_tablet_manager.cc:1428] T b530d8f9feb840e6a5d929dddf550e6d P dbc13a0a54194dbd89fa635b387ddb4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 23:49:44.204241 8982 tablet_bootstrap.cc:492] T b530d8f9feb840e6a5d929dddf550e6d P 8b2255c33ad74a5dae44efa30d85c0a3: No bootstrap required, opened a new log
I20260928 23:49:44.204314 8982 ts_tablet_manager.cc:1397] T b530d8f9feb840e6a5d929dddf550e6d P 8b2255c33ad74a5dae44efa30d85c0a3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:49:44.204622 8983 tablet_bootstrap.cc:492] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964: Bootstrap starting.
I20260928 23:49:44.204705 8982 raft_consensus.cc:348] T b530d8f9feb840e6a5d929dddf550e6d P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:44.204802 8982 raft_consensus.cc:374] T b530d8f9feb840e6a5d929dddf550e6d P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:44.204865 8982 raft_consensus.cc:729] T b530d8f9feb840e6a5d929dddf550e6d P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b2255c33ad74a5dae44efa30d85c0a3, State: Initialized, Role: FOLLOWER
I20260928 23:49:44.204972 8982 consensus_queue.cc:260] T b530d8f9feb840e6a5d929dddf550e6d P 8b2255c33ad74a5dae44efa30d85c0a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:44.205073 9025 tablet_replica.cc:330] T 60c1903310c44123b028bd9f6b9af356 P dbc13a0a54194dbd89fa635b387ddb4e: stopping tablet replica
I20260928 23:49:44.205204 8982 ts_tablet_manager.cc:1428] T b530d8f9feb840e6a5d929dddf550e6d P 8b2255c33ad74a5dae44efa30d85c0a3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:49:44.205358 9025 raft_consensus.cc:2229] T 60c1903310c44123b028bd9f6b9af356 P dbc13a0a54194dbd89fa635b387ddb4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:49:44.205560 9025 raft_consensus.cc:2258] T 60c1903310c44123b028bd9f6b9af356 P dbc13a0a54194dbd89fa635b387ddb4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:49:44.206733 8983 tablet_bootstrap.cc:654] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:44.207291 9025 ts_tablet_manager.cc:1905] T 60c1903310c44123b028bd9f6b9af356 P dbc13a0a54194dbd89fa635b387ddb4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:49:44.207927 8983 tablet_bootstrap.cc:492] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964: No bootstrap required, opened a new log
I20260928 23:49:44.208019 8983 ts_tablet_manager.cc:1397] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 23:49:44.208439 8983 raft_consensus.cc:348] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:44.208535 8983 raft_consensus.cc:374] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:44.208564 8983 raft_consensus.cc:729] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab54842e038346e2b182f5274c66d964, State: Initialized, Role: FOLLOWER
I20260928 23:49:44.208671 8983 consensus_queue.cc:260] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:44.208963 8983 ts_tablet_manager.cc:1428] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:49:44.211287 9025 ts_tablet_manager.cc:1918] T 60c1903310c44123b028bd9f6b9af356 P dbc13a0a54194dbd89fa635b387ddb4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:49:44.211404 9025 log.cc:1196] T 60c1903310c44123b028bd9f6b9af356 P dbc13a0a54194dbd89fa635b387ddb4e: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-0/wal/wals/60c1903310c44123b028bd9f6b9af356
I20260928 23:49:44.211741 9025 ts_tablet_manager.cc:1939] T 60c1903310c44123b028bd9f6b9af356 P dbc13a0a54194dbd89fa635b387ddb4e: Deleting consensus metadata
I20260928 23:49:44.212988 8462 catalog_manager.cc:4841] TS dbc13a0a54194dbd89fa635b387ddb4e (127.6.152.1:43429): tablet 60c1903310c44123b028bd9f6b9af356 (table test-table [id=225a10717885469a9f18c69c39ce2321]) successfully deleted
I20260928 23:49:44.291484 9021 raft_consensus.cc:482] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:49:44.291638 9021 raft_consensus.cc:504] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:44.292078 9021 leader_election.cc:288] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbc13a0a54194dbd89fa635b387ddb4e (127.6.152.1:43429), 8b2255c33ad74a5dae44efa30d85c0a3 (127.6.152.2:41243)
I20260928 23:49:44.292484 8633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b530d8f9feb840e6a5d929dddf550e6d" candidate_uuid: "ab54842e038346e2b182f5274c66d964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" is_pre_election: true
I20260928 23:49:44.292497 8773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b530d8f9feb840e6a5d929dddf550e6d" candidate_uuid: "ab54842e038346e2b182f5274c66d964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" is_pre_election: true
I20260928 23:49:44.292655 8773 raft_consensus.cc:2454] T b530d8f9feb840e6a5d929dddf550e6d P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab54842e038346e2b182f5274c66d964 in term 0.
I20260928 23:49:44.292659 8633 raft_consensus.cc:2454] T b530d8f9feb840e6a5d929dddf550e6d P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab54842e038346e2b182f5274c66d964 in term 0.
I20260928 23:49:44.292953 8847 leader_election.cc:302] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b2255c33ad74a5dae44efa30d85c0a3, ab54842e038346e2b182f5274c66d964; no voters:
I20260928 23:49:44.293115 9021 raft_consensus.cc:2781] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:49:44.293177 9021 raft_consensus.cc:482] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:49:44.293247 9021 raft_consensus.cc:3037] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:44.294423 9021 raft_consensus.cc:504] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:44.294780 9021 leader_election.cc:288] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964 [CANDIDATE]: Term 1 election: Requested vote from peers dbc13a0a54194dbd89fa635b387ddb4e (127.6.152.1:43429), 8b2255c33ad74a5dae44efa30d85c0a3 (127.6.152.2:41243)
I20260928 23:49:44.295113 8633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b530d8f9feb840e6a5d929dddf550e6d" candidate_uuid: "ab54842e038346e2b182f5274c66d964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc13a0a54194dbd89fa635b387ddb4e"
I20260928 23:49:44.295162 8773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b530d8f9feb840e6a5d929dddf550e6d" candidate_uuid: "ab54842e038346e2b182f5274c66d964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b2255c33ad74a5dae44efa30d85c0a3"
I20260928 23:49:44.295246 8633 raft_consensus.cc:3037] T b530d8f9feb840e6a5d929dddf550e6d P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:44.295312 8773 raft_consensus.cc:3037] T b530d8f9feb840e6a5d929dddf550e6d P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:44.296293 8633 raft_consensus.cc:2454] T b530d8f9feb840e6a5d929dddf550e6d P dbc13a0a54194dbd89fa635b387ddb4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab54842e038346e2b182f5274c66d964 in term 1.
I20260928 23:49:44.296432 8773 raft_consensus.cc:2454] T b530d8f9feb840e6a5d929dddf550e6d P 8b2255c33ad74a5dae44efa30d85c0a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab54842e038346e2b182f5274c66d964 in term 1.
I20260928 23:49:44.296665 8846 leader_election.cc:302] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab54842e038346e2b182f5274c66d964, dbc13a0a54194dbd89fa635b387ddb4e; no voters:
I20260928 23:49:44.296888 9021 raft_consensus.cc:2781] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:49:44.296989 9021 raft_consensus.cc:686] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964 [term 1 LEADER]: Becoming Leader. State: Replica: ab54842e038346e2b182f5274c66d964, State: Running, Role: LEADER
I20260928 23:49:44.297137 9021 consensus_queue.cc:237] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:44.298820 8474 catalog_manager.cc:5495] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab54842e038346e2b182f5274c66d964 (127.6.152.3). New cstate: current_term: 1 leader_uuid: "ab54842e038346e2b182f5274c66d964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } health_report { overall_health: HEALTHY } } }
I20260928 23:49:44.318934 8633 raft_consensus.cc:1261] T b530d8f9feb840e6a5d929dddf550e6d P dbc13a0a54194dbd89fa635b387ddb4e [term 1 FOLLOWER]: Refusing update from remote peer ab54842e038346e2b182f5274c66d964: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:49:44.318948 8773 raft_consensus.cc:1261] T b530d8f9feb840e6a5d929dddf550e6d P 8b2255c33ad74a5dae44efa30d85c0a3 [term 1 FOLLOWER]: Refusing update from remote peer ab54842e038346e2b182f5274c66d964: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:49:44.319299 9021 consensus_queue.cc:1035] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:49:44.319482 8992 consensus_queue.cc:1035] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 23:49:44.334651 9007 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 23:49:45.899537 8473 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:49:45.899708 8473 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:49:45.902058 8473 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c: Sending DeleteTablet for 3 replicas of tablet b530d8f9feb840e6a5d929dddf550e6d
I20260928 23:49:45.902693 8613 tablet_service.cc:1514] Processing DeleteTablet for tablet b530d8f9feb840e6a5d929dddf550e6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:49:45 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:60812
I20260928 23:49:45.902740 8893 tablet_service.cc:1514] Processing DeleteTablet for tablet b530d8f9feb840e6a5d929dddf550e6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:49:45 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:44864
I20260928 23:49:45.903054 9051 tablet_replica.cc:330] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964: stopping tablet replica
I20260928 23:49:45.903049 8753 tablet_service.cc:1514] Processing DeleteTablet for tablet b530d8f9feb840e6a5d929dddf550e6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:49:45 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:38136
I20260928 23:49:45.903203 9051 raft_consensus.cc:2229] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:49:45.903426 9051 raft_consensus.cc:2258] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:49:45.903661 9052 tablet_replica.cc:330] T b530d8f9feb840e6a5d929dddf550e6d P dbc13a0a54194dbd89fa635b387ddb4e: stopping tablet replica
I20260928 23:49:45.903782 9052 raft_consensus.cc:2229] T b530d8f9feb840e6a5d929dddf550e6d P dbc13a0a54194dbd89fa635b387ddb4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:49:45.903895 9052 raft_consensus.cc:2258] T b530d8f9feb840e6a5d929dddf550e6d P dbc13a0a54194dbd89fa635b387ddb4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:49:45.904069 9053 tablet_replica.cc:330] T b530d8f9feb840e6a5d929dddf550e6d P 8b2255c33ad74a5dae44efa30d85c0a3: stopping tablet replica
I20260928 23:49:45.904237 9053 raft_consensus.cc:2229] T b530d8f9feb840e6a5d929dddf550e6d P 8b2255c33ad74a5dae44efa30d85c0a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:49:45.904364 9053 raft_consensus.cc:2258] T b530d8f9feb840e6a5d929dddf550e6d P 8b2255c33ad74a5dae44efa30d85c0a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:49:45.904408 9051 ts_tablet_manager.cc:1905] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:49:45.904933 9053 ts_tablet_manager.cc:1905] T b530d8f9feb840e6a5d929dddf550e6d P 8b2255c33ad74a5dae44efa30d85c0a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:49:45.904953 9052 ts_tablet_manager.cc:1905] T b530d8f9feb840e6a5d929dddf550e6d P dbc13a0a54194dbd89fa635b387ddb4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:49:45.907462 9052 ts_tablet_manager.cc:1918] T b530d8f9feb840e6a5d929dddf550e6d P dbc13a0a54194dbd89fa635b387ddb4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 23:49:45.907462 9053 ts_tablet_manager.cc:1918] T b530d8f9feb840e6a5d929dddf550e6d P 8b2255c33ad74a5dae44efa30d85c0a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 23:49:45.907543 9053 log.cc:1196] T b530d8f9feb840e6a5d929dddf550e6d P 8b2255c33ad74a5dae44efa30d85c0a3: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-1/wal/wals/b530d8f9feb840e6a5d929dddf550e6d
I20260928 23:49:45.907543 9052 log.cc:1196] T b530d8f9feb840e6a5d929dddf550e6d P dbc13a0a54194dbd89fa635b387ddb4e: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-0/wal/wals/b530d8f9feb840e6a5d929dddf550e6d
I20260928 23:49:45.907805 9052 ts_tablet_manager.cc:1939] T b530d8f9feb840e6a5d929dddf550e6d P dbc13a0a54194dbd89fa635b387ddb4e: Deleting consensus metadata
I20260928 23:49:45.907805 9053 ts_tablet_manager.cc:1939] T b530d8f9feb840e6a5d929dddf550e6d P 8b2255c33ad74a5dae44efa30d85c0a3: Deleting consensus metadata
I20260928 23:49:45.908512 8463 catalog_manager.cc:4841] TS 8b2255c33ad74a5dae44efa30d85c0a3 (127.6.152.2:41243): tablet b530d8f9feb840e6a5d929dddf550e6d (table test-table [id=4483a8d9317747d783902a9a8bfa4708]) successfully deleted
I20260928 23:49:45.908521 8462 catalog_manager.cc:4841] TS dbc13a0a54194dbd89fa635b387ddb4e (127.6.152.1:43429): tablet b530d8f9feb840e6a5d929dddf550e6d (table test-table [id=4483a8d9317747d783902a9a8bfa4708]) successfully deleted
I20260928 23:49:45.911237 9051 ts_tablet_manager.cc:1918] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 23:49:45.911302 9051 log.cc:1196] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/b530d8f9feb840e6a5d929dddf550e6d
I20260928 23:49:45.911576 9051 ts_tablet_manager.cc:1939] T b530d8f9feb840e6a5d929dddf550e6d P ab54842e038346e2b182f5274c66d964: Deleting consensus metadata
I20260928 23:49:45.912236 8463 catalog_manager.cc:4841] TS ab54842e038346e2b182f5274c66d964 (127.6.152.3:33245): tablet b530d8f9feb840e6a5d929dddf550e6d (table test-table [id=4483a8d9317747d783902a9a8bfa4708]) successfully deleted
Sep 28 23:49:45 dist-test-slave-p5bw krb5kdc[8431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639385, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 23:49:45 dist-test-slave-p5bw krb5kdc[8431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639385, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:45.939881 9061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:45.922995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47694 (local address 127.6.152.62:34125)
0928 23:49:45.923307 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:45.923313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:45.923329 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:49:45.923444 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:45.923447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:45.923491 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:45.923572 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:45.924246 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:45.924934 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:45.925770 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:45.926015 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:45.926109 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:45.936289 (+ 10180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:45.936294 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:45.936297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:45.936331 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:45.937964 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:45.938397 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:45.938402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:45.938408 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:45.938489 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:45.938811 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:45.938815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:45.938846 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:45.939053 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:45.939168 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:45.939469 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 23:49:45.939665 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":107,"threads_started":1}
I20260928 23:49:45.942724 8473 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:49:45.943161 8473 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:49:45.948141 8753 tablet_service.cc:1467] Processing CreateTablet for tablet 069015536041408f9e281ed3d67702f1 (DEFAULT_TABLE table=test-table [id=baf6a68797e244908de219ba65b94c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:45.948141 8613 tablet_service.cc:1467] Processing CreateTablet for tablet 069015536041408f9e281ed3d67702f1 (DEFAULT_TABLE table=test-table [id=baf6a68797e244908de219ba65b94c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:45.948236 8893 tablet_service.cc:1467] Processing CreateTablet for tablet 069015536041408f9e281ed3d67702f1 (DEFAULT_TABLE table=test-table [id=baf6a68797e244908de219ba65b94c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:45.948441 8613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 069015536041408f9e281ed3d67702f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:45.948441 8753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 069015536041408f9e281ed3d67702f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:45.948556 8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 069015536041408f9e281ed3d67702f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:45.951081 9064 tablet_bootstrap.cc:492] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e: Bootstrap starting.
I20260928 23:49:45.951115 9062 tablet_bootstrap.cc:492] T 069015536041408f9e281ed3d67702f1 P ab54842e038346e2b182f5274c66d964: Bootstrap starting.
I20260928 23:49:45.952096 9062 tablet_bootstrap.cc:654] T 069015536041408f9e281ed3d67702f1 P ab54842e038346e2b182f5274c66d964: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:45.952409 9063 tablet_bootstrap.cc:492] T 069015536041408f9e281ed3d67702f1 P 8b2255c33ad74a5dae44efa30d85c0a3: Bootstrap starting.
I20260928 23:49:45.952421 9064 tablet_bootstrap.cc:654] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:45.953497 9062 tablet_bootstrap.cc:492] T 069015536041408f9e281ed3d67702f1 P ab54842e038346e2b182f5274c66d964: No bootstrap required, opened a new log
I20260928 23:49:45.953573 9063 tablet_bootstrap.cc:654] T 069015536041408f9e281ed3d67702f1 P 8b2255c33ad74a5dae44efa30d85c0a3: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:45.953586 9062 ts_tablet_manager.cc:1397] T 069015536041408f9e281ed3d67702f1 P ab54842e038346e2b182f5274c66d964: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:49:45.953675 9064 tablet_bootstrap.cc:492] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e: No bootstrap required, opened a new log
I20260928 23:49:45.953750 9064 ts_tablet_manager.cc:1397] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:49:45.954016 9062 raft_consensus.cc:348] T 069015536041408f9e281ed3d67702f1 P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:45.954138 9062 raft_consensus.cc:374] T 069015536041408f9e281ed3d67702f1 P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:45.954186 9062 raft_consensus.cc:729] T 069015536041408f9e281ed3d67702f1 P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab54842e038346e2b182f5274c66d964, State: Initialized, Role: FOLLOWER
I20260928 23:49:45.954159 9064 raft_consensus.cc:348] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:45.954282 9064 raft_consensus.cc:374] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:45.954314 9064 raft_consensus.cc:729] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbc13a0a54194dbd89fa635b387ddb4e, State: Initialized, Role: FOLLOWER
I20260928 23:49:45.954336 9062 consensus_queue.cc:260] T 069015536041408f9e281ed3d67702f1 P ab54842e038346e2b182f5274c66d964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:45.954437 9064 consensus_queue.cc:260] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:45.954643 9062 ts_tablet_manager.cc:1428] T 069015536041408f9e281ed3d67702f1 P ab54842e038346e2b182f5274c66d964: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:49:45.954806 9064 ts_tablet_manager.cc:1428] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:49:45.955410 9063 tablet_bootstrap.cc:492] T 069015536041408f9e281ed3d67702f1 P 8b2255c33ad74a5dae44efa30d85c0a3: No bootstrap required, opened a new log
I20260928 23:49:45.955499 9063 ts_tablet_manager.cc:1397] T 069015536041408f9e281ed3d67702f1 P 8b2255c33ad74a5dae44efa30d85c0a3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:49:45.955997 9063 raft_consensus.cc:348] T 069015536041408f9e281ed3d67702f1 P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:45.956136 9063 raft_consensus.cc:374] T 069015536041408f9e281ed3d67702f1 P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:45.956168 9063 raft_consensus.cc:729] T 069015536041408f9e281ed3d67702f1 P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b2255c33ad74a5dae44efa30d85c0a3, State: Initialized, Role: FOLLOWER
I20260928 23:49:45.956293 9063 consensus_queue.cc:260] T 069015536041408f9e281ed3d67702f1 P 8b2255c33ad74a5dae44efa30d85c0a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:45.956697 9063 ts_tablet_manager.cc:1428] T 069015536041408f9e281ed3d67702f1 P 8b2255c33ad74a5dae44efa30d85c0a3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:49:46.287389 9068 raft_consensus.cc:482] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:49:46.287549 9068 raft_consensus.cc:504] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:46.288558 9068 leader_election.cc:288] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b2255c33ad74a5dae44efa30d85c0a3 (127.6.152.2:41243), ab54842e038346e2b182f5274c66d964 (127.6.152.3:33245)
I20260928 23:49:46.292541 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:46.288945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:35099 (local address 127.6.152.2:41243)
0928 23:49:46.289280 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:46.289288 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:46.289304 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:49:46.289455 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:46.289459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:46.289515 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:46.289625 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:49:46.289904 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.290452 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.291529 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.291946 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.292082 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:46.292146 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:46.292323 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 23:49:46.292393 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":129,"threads_started":1}
I20260928 23:49:46.292547 9070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:46.288872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:41243 (local address 127.6.152.1:35099)
0928 23:49:46.289166 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:46.289179 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:46.289332 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:46.289638 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:46.289641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:46.289659 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:49:46.289789 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:46.289794 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.290573 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.290576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:46.291408 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:46.291417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.292090 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.292094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:46.292190 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:46.292212 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:49:46.292309 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 23:49:46.292377 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":125,"threads_started":1}
I20260928 23:49:46.292765 9073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:46.289406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:55207 (local address 127.6.152.3:33245)
0928 23:49:46.289693 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:46.289701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:46.289715 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:49:46.289795 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:46.289797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:46.289878 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:46.289971 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:49:46.290260 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.290767 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.291822 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.292229 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.292347 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:46.292407 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:46.292581 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 23:49:46.292651 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":107,"threads_started":1}
I20260928 23:49:46.292776 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:46.289296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:33245 (local address 127.6.152.1:55207)
0928 23:49:46.289600 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:46.289615 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:46.289713 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:46.289995 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:46.289998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:46.290014 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:49:46.290140 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:46.290145 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.290894 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.290897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:46.291686 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:46.291695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.292352 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.292355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:46.292460 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:46.292478 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 23:49:46.292565 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 23:49:46.292632 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":109,"threads_started":1}
I20260928 23:49:46.293434 8772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069015536041408f9e281ed3d67702f1" candidate_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" is_pre_election: true
I20260928 23:49:46.293481 8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069015536041408f9e281ed3d67702f1" candidate_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab54842e038346e2b182f5274c66d964" is_pre_election: true
I20260928 23:49:46.293627 8772 raft_consensus.cc:2454] T 069015536041408f9e281ed3d67702f1 P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbc13a0a54194dbd89fa635b387ddb4e in term 0.
I20260928 23:49:46.293742 8913 raft_consensus.cc:2454] T 069015536041408f9e281ed3d67702f1 P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbc13a0a54194dbd89fa635b387ddb4e in term 0.
I20260928 23:49:46.293987 8567 leader_election.cc:302] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b2255c33ad74a5dae44efa30d85c0a3, dbc13a0a54194dbd89fa635b387ddb4e; no voters:
I20260928 23:49:46.294238 9068 raft_consensus.cc:2781] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:49:46.294296 9068 raft_consensus.cc:482] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:49:46.294324 9068 raft_consensus.cc:3037] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:46.295485 9068 raft_consensus.cc:504] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:46.295828 9068 leader_election.cc:288] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e [CANDIDATE]: Term 1 election: Requested vote from peers 8b2255c33ad74a5dae44efa30d85c0a3 (127.6.152.2:41243), ab54842e038346e2b182f5274c66d964 (127.6.152.3:33245)
I20260928 23:49:46.296233 8772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069015536041408f9e281ed3d67702f1" candidate_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b2255c33ad74a5dae44efa30d85c0a3"
I20260928 23:49:46.296304 8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069015536041408f9e281ed3d67702f1" candidate_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab54842e038346e2b182f5274c66d964"
I20260928 23:49:46.296350 8772 raft_consensus.cc:3037] T 069015536041408f9e281ed3d67702f1 P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:46.296411 8913 raft_consensus.cc:3037] T 069015536041408f9e281ed3d67702f1 P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:46.297544 8772 raft_consensus.cc:2454] T 069015536041408f9e281ed3d67702f1 P 8b2255c33ad74a5dae44efa30d85c0a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbc13a0a54194dbd89fa635b387ddb4e in term 1.
I20260928 23:49:46.297554 8913 raft_consensus.cc:2454] T 069015536041408f9e281ed3d67702f1 P ab54842e038346e2b182f5274c66d964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbc13a0a54194dbd89fa635b387ddb4e in term 1.
I20260928 23:49:46.297816 8567 leader_election.cc:302] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab54842e038346e2b182f5274c66d964, dbc13a0a54194dbd89fa635b387ddb4e; no voters:
I20260928 23:49:46.298000 9068 raft_consensus.cc:2781] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:49:46.298204 9068 raft_consensus.cc:686] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e [term 1 LEADER]: Becoming Leader. State: Replica: dbc13a0a54194dbd89fa635b387ddb4e, State: Running, Role: LEADER
I20260928 23:49:46.298466 9068 consensus_queue.cc:237] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:46.300696 8476 catalog_manager.cc:5495] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e reported cstate change: term changed from 0 to 1, leader changed from <none> to dbc13a0a54194dbd89fa635b387ddb4e (127.6.152.1). New cstate: current_term: 1 leader_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } health_report { overall_health: UNKNOWN } } }
I20260928 23:49:46.321910 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:46.317959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60818 (local address 127.6.152.1:43429)
0928 23:49:46.318199 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:46.318212 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:46.318225 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:49:46.318295 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:46.318297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:46.318365 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:46.318469 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:49:46.318715 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.319266 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.320203 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.320463 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.320573 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:46.320896 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:49:46.321475 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:49:46.321603 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:46.321665 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 23:49:46.321720 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":103,"threads_started":1}
I20260928 23:49:46.331858 9073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:46.328888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:40503 (local address 127.6.152.3:33245)
0928 23:49:46.329011 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:46.329015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:46.329166 (+ 151us) server_negotiation.cc:407] Connection header received
0928 23:49:46.329245 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:46.329247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:46.329296 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:46.329362 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:49:46.329667 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.330176 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.330984 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.331184 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.331258 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:46.331517 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:49:46.331580 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:49:46.331631 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:46.331710 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 23:49:46.331752 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 23:49:46.331885 9080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:46.328726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:33245 (local address 127.6.152.1:40503)
0928 23:49:46.329072 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:46.329088 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:46.329176 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:46.329418 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:46.329421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:46.329430 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:49:46.329557 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:46.329564 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.330272 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.330275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:46.330878 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:46.330885 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.331289 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.331291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:46.331365 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:46.331406 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:49:46.331651 (+ 245us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:49:46.331655 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:49:46.331704 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 23:49:46.331765 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":159,"threads_started":1}
I20260928 23:49:46.338151 8772 raft_consensus.cc:1261] T 069015536041408f9e281ed3d67702f1 P 8b2255c33ad74a5dae44efa30d85c0a3 [term 1 FOLLOWER]: Refusing update from remote peer dbc13a0a54194dbd89fa635b387ddb4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:49:46.338312 8913 raft_consensus.cc:1261] T 069015536041408f9e281ed3d67702f1 P ab54842e038346e2b182f5274c66d964 [term 1 FOLLOWER]: Refusing update from remote peer dbc13a0a54194dbd89fa635b387ddb4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:49:46.338634 9068 consensus_queue.cc:1035] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:49:46.338924 9074 consensus_queue.cc:1035] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:49:46.360498 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:46.356448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:43429 (local address 127.6.152.3:39615)
0928 23:49:46.356715 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:46.356730 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:46.356869 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:46.357133 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:46.357136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:46.357144 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:49:46.357248 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:46.357256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.358195 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.358199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:46.359201 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:46.359210 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.359768 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.359772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:46.359866 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:46.359892 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:49:46.360186 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:49:46.360193 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 23:49:46.360262 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 23:49:46.360352 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":107,"threads_started":1}
I20260928 23:49:46.360481 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:46.356491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:39615 (local address 127.6.152.1:43429)
0928 23:49:46.356616 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:46.356620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:46.356931 (+ 311us) server_negotiation.cc:407] Connection header received
0928 23:49:46.356986 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:46.356989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:46.357040 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:46.357110 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:49:46.357393 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.358093 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.359319 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.359654 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.359771 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:46.360015 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:49:46.360097 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:49:46.360170 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:46.360291 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 23:49:46.360356 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 23:49:46.389257 8473 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:49:46.389447 8473 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:49:46.391567 8473 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c: Sending DeleteTablet for 3 replicas of tablet 069015536041408f9e281ed3d67702f1
I20260928 23:49:46.392167 8613 tablet_service.cc:1514] Processing DeleteTablet for tablet 069015536041408f9e281ed3d67702f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:49:46 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:60812
I20260928 23:49:46.392359 9052 tablet_replica.cc:330] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e: stopping tablet replica
I20260928 23:49:46.392351 8753 tablet_service.cc:1514] Processing DeleteTablet for tablet 069015536041408f9e281ed3d67702f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:49:46 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:38136
I20260928 23:49:46.392472 9052 raft_consensus.cc:2229] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:49:46.392794 9052 raft_consensus.cc:2258] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:49:46.392881 8893 tablet_service.cc:1514] Processing DeleteTablet for tablet 069015536041408f9e281ed3d67702f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:49:46 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:44864
I20260928 23:49:46.393019 9053 tablet_replica.cc:330] T 069015536041408f9e281ed3d67702f1 P 8b2255c33ad74a5dae44efa30d85c0a3: stopping tablet replica
I20260928 23:49:46.393066 9051 tablet_replica.cc:330] T 069015536041408f9e281ed3d67702f1 P ab54842e038346e2b182f5274c66d964: stopping tablet replica
I20260928 23:49:46.393147 9053 raft_consensus.cc:2229] T 069015536041408f9e281ed3d67702f1 P 8b2255c33ad74a5dae44efa30d85c0a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:49:46.393185 9051 raft_consensus.cc:2229] T 069015536041408f9e281ed3d67702f1 P ab54842e038346e2b182f5274c66d964 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:49:46.393121 8473 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 23:49:46.393258 9053 raft_consensus.cc:2258] T 069015536041408f9e281ed3d67702f1 P 8b2255c33ad74a5dae44efa30d85c0a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:49:46.393297 9051 raft_consensus.cc:2258] T 069015536041408f9e281ed3d67702f1 P ab54842e038346e2b182f5274c66d964 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 23:49:46.393545 8473 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:49:46.394138 9053 ts_tablet_manager.cc:1905] T 069015536041408f9e281ed3d67702f1 P 8b2255c33ad74a5dae44efa30d85c0a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:49:46.394160 9051 ts_tablet_manager.cc:1905] T 069015536041408f9e281ed3d67702f1 P ab54842e038346e2b182f5274c66d964: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:49:46.395443 9052 ts_tablet_manager.cc:1905] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:49:46.399561 8613 tablet_service.cc:1467] Processing CreateTablet for tablet 866894a90cc54718be18c3b707a62aee (DEFAULT_TABLE table=test-table [id=46f3be41b13d4c0ba27d24e738dffd24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:46.399549 8753 tablet_service.cc:1467] Processing CreateTablet for tablet 866894a90cc54718be18c3b707a62aee (DEFAULT_TABLE table=test-table [id=46f3be41b13d4c0ba27d24e738dffd24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:46.399704 8893 tablet_service.cc:1467] Processing CreateTablet for tablet 866894a90cc54718be18c3b707a62aee (DEFAULT_TABLE table=test-table [id=46f3be41b13d4c0ba27d24e738dffd24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:46.405360 8753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866894a90cc54718be18c3b707a62aee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:46.407388 9063 tablet_bootstrap.cc:492] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3: Bootstrap starting.
I20260928 23:49:46.408116 8613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866894a90cc54718be18c3b707a62aee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:46.408396 9063 tablet_bootstrap.cc:654] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:46.409746 9063 tablet_bootstrap.cc:492] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3: No bootstrap required, opened a new log
I20260928 23:49:46.409830 9063 ts_tablet_manager.cc:1397] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:49:46.410048 8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866894a90cc54718be18c3b707a62aee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:46.410270 9063 raft_consensus.cc:348] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:46.410403 9063 raft_consensus.cc:374] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:46.410439 9063 raft_consensus.cc:729] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b2255c33ad74a5dae44efa30d85c0a3, State: Initialized, Role: FOLLOWER
I20260928 23:49:46.410557 9064 tablet_bootstrap.cc:492] T 866894a90cc54718be18c3b707a62aee P dbc13a0a54194dbd89fa635b387ddb4e: Bootstrap starting.
I20260928 23:49:46.410576 9063 consensus_queue.cc:260] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:46.410863 9063 ts_tablet_manager.cc:1428] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:49:46.411517 9064 tablet_bootstrap.cc:654] T 866894a90cc54718be18c3b707a62aee P dbc13a0a54194dbd89fa635b387ddb4e: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:46.412712 9064 tablet_bootstrap.cc:492] T 866894a90cc54718be18c3b707a62aee P dbc13a0a54194dbd89fa635b387ddb4e: No bootstrap required, opened a new log
I20260928 23:49:46.412827 9064 ts_tablet_manager.cc:1397] T 866894a90cc54718be18c3b707a62aee P dbc13a0a54194dbd89fa635b387ddb4e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:49:46.413249 9064 raft_consensus.cc:348] T 866894a90cc54718be18c3b707a62aee P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:46.413371 9064 raft_consensus.cc:374] T 866894a90cc54718be18c3b707a62aee P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:46.413410 9064 raft_consensus.cc:729] T 866894a90cc54718be18c3b707a62aee P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbc13a0a54194dbd89fa635b387ddb4e, State: Initialized, Role: FOLLOWER
I20260928 23:49:46.413542 9064 consensus_queue.cc:260] T 866894a90cc54718be18c3b707a62aee P dbc13a0a54194dbd89fa635b387ddb4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:46.413787 9064 ts_tablet_manager.cc:1428] T 866894a90cc54718be18c3b707a62aee P dbc13a0a54194dbd89fa635b387ddb4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:49:46.413841 9062 tablet_bootstrap.cc:492] T 866894a90cc54718be18c3b707a62aee P ab54842e038346e2b182f5274c66d964: Bootstrap starting.
I20260928 23:49:46.414644 9062 tablet_bootstrap.cc:654] T 866894a90cc54718be18c3b707a62aee P ab54842e038346e2b182f5274c66d964: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:46.415895 9062 tablet_bootstrap.cc:492] T 866894a90cc54718be18c3b707a62aee P ab54842e038346e2b182f5274c66d964: No bootstrap required, opened a new log
I20260928 23:49:46.416007 9062 ts_tablet_manager.cc:1397] T 866894a90cc54718be18c3b707a62aee P ab54842e038346e2b182f5274c66d964: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:49:46.416446 9062 raft_consensus.cc:348] T 866894a90cc54718be18c3b707a62aee P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:46.416575 9062 raft_consensus.cc:374] T 866894a90cc54718be18c3b707a62aee P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:46.416617 9062 raft_consensus.cc:729] T 866894a90cc54718be18c3b707a62aee P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab54842e038346e2b182f5274c66d964, State: Initialized, Role: FOLLOWER
I20260928 23:49:46.416782 9062 consensus_queue.cc:260] T 866894a90cc54718be18c3b707a62aee P ab54842e038346e2b182f5274c66d964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:46.417128 9062 ts_tablet_manager.cc:1428] T 866894a90cc54718be18c3b707a62aee P ab54842e038346e2b182f5274c66d964: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:49:46.418167 9053 ts_tablet_manager.cc:1918] T 069015536041408f9e281ed3d67702f1 P 8b2255c33ad74a5dae44efa30d85c0a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:49:46.418362 9053 log.cc:1196] T 069015536041408f9e281ed3d67702f1 P 8b2255c33ad74a5dae44efa30d85c0a3: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-1/wal/wals/069015536041408f9e281ed3d67702f1
I20260928 23:49:46.418660 9053 ts_tablet_manager.cc:1939] T 069015536041408f9e281ed3d67702f1 P 8b2255c33ad74a5dae44efa30d85c0a3: Deleting consensus metadata
I20260928 23:49:46.427476 8463 catalog_manager.cc:4841] TS 8b2255c33ad74a5dae44efa30d85c0a3 (127.6.152.2:41243): tablet 069015536041408f9e281ed3d67702f1 (table test-table [id=baf6a68797e244908de219ba65b94c7e]) successfully deleted
I20260928 23:49:46.478725 9052 ts_tablet_manager.cc:1918] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:49:46.478727 9051 ts_tablet_manager.cc:1918] T 069015536041408f9e281ed3d67702f1 P ab54842e038346e2b182f5274c66d964: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:49:46.478853 9052 log.cc:1196] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-0/wal/wals/069015536041408f9e281ed3d67702f1
I20260928 23:49:46.478869 9051 log.cc:1196] T 069015536041408f9e281ed3d67702f1 P ab54842e038346e2b182f5274c66d964: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/069015536041408f9e281ed3d67702f1
I20260928 23:49:46.479151 9052 ts_tablet_manager.cc:1939] T 069015536041408f9e281ed3d67702f1 P dbc13a0a54194dbd89fa635b387ddb4e: Deleting consensus metadata
I20260928 23:49:46.479154 9051 ts_tablet_manager.cc:1939] T 069015536041408f9e281ed3d67702f1 P ab54842e038346e2b182f5274c66d964: Deleting consensus metadata
I20260928 23:49:46.480178 8463 catalog_manager.cc:4841] TS ab54842e038346e2b182f5274c66d964 (127.6.152.3:33245): tablet 069015536041408f9e281ed3d67702f1 (table test-table [id=baf6a68797e244908de219ba65b94c7e]) successfully deleted
I20260928 23:49:46.490285 9069 raft_consensus.cc:482] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:49:46.490429 9069 raft_consensus.cc:504] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:46.491542 9069 leader_election.cc:288] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbc13a0a54194dbd89fa635b387ddb4e (127.6.152.1:43429), ab54842e038346e2b182f5274c66d964 (127.6.152.3:33245)
I20260928 23:49:46.496878 9073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:46.491893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:50977 (local address 127.6.152.3:33245)
0928 23:49:46.492022 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:46.492027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:46.492229 (+ 202us) server_negotiation.cc:407] Connection header received
0928 23:49:46.492518 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:46.492521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:46.492580 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:46.492665 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:49:46.493196 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.493804 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.495593 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.496269 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.496381 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:46.496470 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:46.496618 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 23:49:46.496698 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:49:46.497205 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:46.491767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:33245 (local address 127.6.152.2:50977)
0928 23:49:46.492129 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:46.492140 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:46.492276 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:46.492994 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:46.492998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:46.493013 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:49:46.493104 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:46.493113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.493933 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.493936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:46.495456 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:46.495466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.496405 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.496408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:46.496483 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:46.496504 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 23:49:46.496597 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 23:49:46.496674 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":24,"thread_start_us":120,"threads_started":1}
I20260928 23:49:46.497686 8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866894a90cc54718be18c3b707a62aee" candidate_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab54842e038346e2b182f5274c66d964" is_pre_election: true
I20260928 23:49:46.497867 8913 raft_consensus.cc:2454] T 866894a90cc54718be18c3b707a62aee P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b2255c33ad74a5dae44efa30d85c0a3 in term 0.
I20260928 23:49:46.497946 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:46.491765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:43429 (local address 127.6.152.2:34783)
0928 23:49:46.492469 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:46.492488 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:46.492597 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:46.492887 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:46.492890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:46.492905 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:49:46.493031 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:46.493038 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.494091 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.494094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:46.495458 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:46.495466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.497518 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.497522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:46.497631 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:46.497654 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:49:46.497732 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 23:49:46.497802 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":411,"thread_start_us":122,"threads_started":1}
I20260928 23:49:46.497974 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:46.491942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:34783 (local address 127.6.152.1:43429)
0928 23:49:46.492112 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:46.492118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:46.492570 (+ 452us) server_negotiation.cc:407] Connection header received
0928 23:49:46.492703 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:46.492707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:46.492762 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:46.492874 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:49:46.493158 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.493969 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.496536 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.497294 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.497398 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:46.497459 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:46.497769 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 23:49:46.497858 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 23:49:46.498396 8632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866894a90cc54718be18c3b707a62aee" candidate_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" is_pre_election: true
I20260928 23:49:46.498553 8632 raft_consensus.cc:2454] T 866894a90cc54718be18c3b707a62aee P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b2255c33ad74a5dae44efa30d85c0a3 in term 0.
I20260928 23:49:46.498660 8707 leader_election.cc:302] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b2255c33ad74a5dae44efa30d85c0a3, ab54842e038346e2b182f5274c66d964; no voters:
I20260928 23:49:46.499167 9069 raft_consensus.cc:2781] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:49:46.499241 9069 raft_consensus.cc:482] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:49:46.499291 9069 raft_consensus.cc:3037] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:46.499356 8462 catalog_manager.cc:4841] TS dbc13a0a54194dbd89fa635b387ddb4e (127.6.152.1:43429): tablet 069015536041408f9e281ed3d67702f1 (table test-table [id=baf6a68797e244908de219ba65b94c7e]) successfully deleted
I20260928 23:49:46.578611 9069 raft_consensus.cc:504] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:46.579164 9069 leader_election.cc:288] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3 [CANDIDATE]: Term 1 election: Requested vote from peers dbc13a0a54194dbd89fa635b387ddb4e (127.6.152.1:43429), ab54842e038346e2b182f5274c66d964 (127.6.152.3:33245)
I20260928 23:49:46.579676 8913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866894a90cc54718be18c3b707a62aee" candidate_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab54842e038346e2b182f5274c66d964"
I20260928 23:49:46.579739 8632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866894a90cc54718be18c3b707a62aee" candidate_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc13a0a54194dbd89fa635b387ddb4e"
I20260928 23:49:46.579854 8913 raft_consensus.cc:3037] T 866894a90cc54718be18c3b707a62aee P ab54842e038346e2b182f5274c66d964 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:46.579879 8632 raft_consensus.cc:3037] T 866894a90cc54718be18c3b707a62aee P dbc13a0a54194dbd89fa635b387ddb4e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:46.581180 8632 raft_consensus.cc:2454] T 866894a90cc54718be18c3b707a62aee P dbc13a0a54194dbd89fa635b387ddb4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b2255c33ad74a5dae44efa30d85c0a3 in term 1.
I20260928 23:49:46.581378 8913 raft_consensus.cc:2454] T 866894a90cc54718be18c3b707a62aee P ab54842e038346e2b182f5274c66d964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b2255c33ad74a5dae44efa30d85c0a3 in term 1.
I20260928 23:49:46.581561 8706 leader_election.cc:302] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b2255c33ad74a5dae44efa30d85c0a3, dbc13a0a54194dbd89fa635b387ddb4e; no voters:
I20260928 23:49:46.581751 9069 raft_consensus.cc:2781] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:49:46.581946 9069 raft_consensus.cc:686] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3 [term 1 LEADER]: Becoming Leader. State: Replica: 8b2255c33ad74a5dae44efa30d85c0a3, State: Running, Role: LEADER
I20260928 23:49:46.582212 9069 consensus_queue.cc:237] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } }
I20260928 23:49:46.585243 8473 catalog_manager.cc:5495] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b2255c33ad74a5dae44efa30d85c0a3 (127.6.152.2). New cstate: current_term: 1 leader_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b2255c33ad74a5dae44efa30d85c0a3" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 } health_report { overall_health: UNKNOWN } } }
I20260928 23:49:46.608515 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:46.604844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.6.152.2:41243)
0928 23:49:46.604948 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:46.604952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:46.605050 (+ 98us) server_negotiation.cc:407] Connection header received
0928 23:49:46.605222 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:46.605225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:46.605288 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:46.605401 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:49:46.605664 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.606183 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.607024 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.607258 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.607378 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:46.607652 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:49:46.608121 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:49:46.608251 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:46.608309 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 23:49:46.608366 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 23:49:46.618251 9073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:46.615304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:50137 (local address 127.6.152.3:33245)
0928 23:49:46.615431 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:46.615434 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:46.615461 (+ 27us) server_negotiation.cc:407] Connection header received
0928 23:49:46.615583 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:46.615585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:46.615626 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:46.615688 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:49:46.615896 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.616409 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.617333 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.617571 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.617649 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:46.617885 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:49:46.617954 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:49:46.618007 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:46.618103 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 23:49:46.618147 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 23:49:46.618278 9096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:46.615114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:33245 (local address 127.6.152.2:50137)
0928 23:49:46.615405 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:46.615418 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:46.615499 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:46.615712 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:46.615714 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:46.615721 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:49:46.615826 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:46.615830 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.616521 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.616525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:46.617232 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:46.617242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.617662 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.617665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:46.617732 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:46.617772 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:49:46.618034 (+ 262us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:49:46.618038 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:49:46.618087 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 23:49:46.618146 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":109,"threads_started":1}
I20260928 23:49:46.624191 8913 raft_consensus.cc:1261] T 866894a90cc54718be18c3b707a62aee P ab54842e038346e2b182f5274c66d964 [term 1 FOLLOWER]: Refusing update from remote peer 8b2255c33ad74a5dae44efa30d85c0a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:49:46.624172 8632 raft_consensus.cc:1261] T 866894a90cc54718be18c3b707a62aee P dbc13a0a54194dbd89fa635b387ddb4e [term 1 FOLLOWER]: Refusing update from remote peer 8b2255c33ad74a5dae44efa30d85c0a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:49:46.624675 9092 consensus_queue.cc:1035] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab54842e038346e2b182f5274c66d964" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:49:46.624888 9069 consensus_queue.cc:1035] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbc13a0a54194dbd89fa635b387ddb4e" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 43429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:49:46.642148 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:46.638688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:41243 (local address 127.6.152.3:48623)
0928 23:49:46.638829 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:46.638845 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:46.638947 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:46.639222 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:46.639225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:46.639232 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:49:46.639333 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:46.639338 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.640238 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.640244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:46.640920 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:46.640937 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.641417 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.641420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:46.641523 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:46.641561 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:49:46.641846 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:49:46.641854 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 23:49:46.641915 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 23:49:46.641997 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 23:49:46.642132 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:46.638726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:48623 (local address 127.6.152.2:41243)
0928 23:49:46.638828 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:46.638833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:46.638897 (+ 64us) server_negotiation.cc:407] Connection header received
0928 23:49:46.639060 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:46.639064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:46.639118 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:46.639212 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:49:46.639443 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.640134 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.641083 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:46.641310 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:46.641418 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:46.641678 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:49:46.641758 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:49:46.641832 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:46.641926 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 23:49:46.641977 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 23:49:46.643062 9035 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 23:49:48.200337 8474 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:49:48.200524 8474 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:49:48.202903 8474 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a917c45eee74d01830e12a3fc1b8f6c: Sending DeleteTablet for 3 replicas of tablet 866894a90cc54718be18c3b707a62aee
I20260928 23:49:48.203454 8613 tablet_service.cc:1514] Processing DeleteTablet for tablet 866894a90cc54718be18c3b707a62aee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:49:48 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:60812
I20260928 23:49:48.203598 8893 tablet_service.cc:1514] Processing DeleteTablet for tablet 866894a90cc54718be18c3b707a62aee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:49:48 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:44864
I20260928 23:49:48.203600 8753 tablet_service.cc:1514] Processing DeleteTablet for tablet 866894a90cc54718be18c3b707a62aee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:49:48 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:38136
I20260928 23:49:48.203979 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 8548
I20260928 23:49:48.204109 9125 tablet_replica.cc:330] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3: stopping tablet replica
I20260928 23:49:48.205983 9125 raft_consensus.cc:2229] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:49:48.206327 9125 raft_consensus.cc:2258] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:49:48.207024 9125 ts_tablet_manager.cc:1905] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:49:48.208990 9124 tablet_replica.cc:330] T 866894a90cc54718be18c3b707a62aee P ab54842e038346e2b182f5274c66d964: stopping tablet replica
I20260928 23:49:48.209151 9124 raft_consensus.cc:2229] T 866894a90cc54718be18c3b707a62aee P ab54842e038346e2b182f5274c66d964 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:49:48.209276 9124 raft_consensus.cc:2258] T 866894a90cc54718be18c3b707a62aee P ab54842e038346e2b182f5274c66d964 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:49:48.209949 9124 ts_tablet_manager.cc:1905] T 866894a90cc54718be18c3b707a62aee P ab54842e038346e2b182f5274c66d964: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:49:48.210781 9125 ts_tablet_manager.cc:1918] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 23:49:48.210881 9125 log.cc:1196] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-1/wal/wals/866894a90cc54718be18c3b707a62aee
I20260928 23:49:48.211277 9125 ts_tablet_manager.cc:1939] T 866894a90cc54718be18c3b707a62aee P 8b2255c33ad74a5dae44efa30d85c0a3: Deleting consensus metadata
I20260928 23:49:48.212148 9124 ts_tablet_manager.cc:1918] T 866894a90cc54718be18c3b707a62aee P ab54842e038346e2b182f5274c66d964: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 23:49:48.212112 8463 catalog_manager.cc:4841] TS 8b2255c33ad74a5dae44efa30d85c0a3 (127.6.152.2:41243): tablet 866894a90cc54718be18c3b707a62aee (table test-table [id=46f3be41b13d4c0ba27d24e738dffd24]) successfully deleted
I20260928 23:49:48.212240 9124 log.cc:1196] T 866894a90cc54718be18c3b707a62aee P ab54842e038346e2b182f5274c66d964: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/866894a90cc54718be18c3b707a62aee
I20260928 23:49:48.212487 9124 ts_tablet_manager.cc:1939] T 866894a90cc54718be18c3b707a62aee P ab54842e038346e2b182f5274c66d964: Deleting consensus metadata
I20260928 23:49:48.213188 8463 catalog_manager.cc:4841] TS ab54842e038346e2b182f5274c66d964 (127.6.152.3:33245): tablet 866894a90cc54718be18c3b707a62aee (table test-table [id=46f3be41b13d4c0ba27d24e738dffd24]) successfully deleted
W20260928 23:49:48.213639 8462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.152.1:43429 (error 108)
I20260928 23:49:48.213770 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 8688
I20260928 23:49:48.215426 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:48.214921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:43429 (local address 127.0.0.1:60828)
0928 23:49:48.215186 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:48.215278 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.1:43429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":68,"threads_started":1}
W20260928 23:49:48.215561 8462 catalog_manager.cc:4584] TS dbc13a0a54194dbd89fa635b387ddb4e (127.6.152.1:43429): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 866894a90cc54718be18c3b707a62aee: Network error: Client connection negotiation failed: client connection to 127.6.152.1:43429: connect: Connection refused (error 111)
I20260928 23:49:48.222118 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 8828
I20260928 23:49:48.228688 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 8444
2026-09-28T23:49:48Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6516 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:49:48 dist-test-slave-p5bw krb5kdc[9130](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:49:48 dist-test-slave-p5bw krb5kdc[9130](info): set up 2 sockets
Sep 28 23:49:48 dist-test-slave-p5bw krb5kdc[9130](info): commencing operation
krb5kdc: starting...
W20260928 23:49:49.261467 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:49:49 dist-test-slave-p5bw krb5kdc[9130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:49:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:49:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:49:49.383960 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:39831
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:39831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:49.495417 9143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:49.495700 9143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:49.495741 9143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:49.499883 9143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:49:49.499954 9143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:49.499974 9143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:49.499992 9143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:49:49.500007 9143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:49:49.505537 9143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:39831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:39831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:49.506639 9143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:49.507678 9143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:49.516762 9149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:49.516759 9148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:49.516793 9151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:49.517153 9143 server_base.cc:1034] running on GCE node
I20260928 23:49:49.517756 9143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:49.518909 9143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:49.520107 9143 hybrid_clock.cc:648] HybridClock initialized: now 1790639389520078 us; error 52 us; skew 500 ppm
Sep 28 23:49:49 dist-test-slave-p5bw krb5kdc[9130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639389, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:49.522889 9143 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:49:49.523820 9143 webserver.cc:460] Webserver started at http://127.6.152.62:46173/ using document root <none> and password file <none>
I20260928 23:49:49.524394 9143 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:49.524485 9143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:49.524720 9143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:49.526827 9143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "ce422dbbe094457b990271ea6891b8e9"
format_stamp: "Formatted at 2026-09-28 23:49:49 on dist-test-slave-p5bw"
server_key: "f9f2258aaf820be627b9a4bb9effbe48"
server_key_iv: "55ff8119da0b9ab8893ef9463f0baeff"
server_key_version: "encryptionkey@0"
I20260928 23:49:49.527393 9143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "ce422dbbe094457b990271ea6891b8e9"
format_stamp: "Formatted at 2026-09-28 23:49:49 on dist-test-slave-p5bw"
server_key: "f9f2258aaf820be627b9a4bb9effbe48"
server_key_iv: "55ff8119da0b9ab8893ef9463f0baeff"
server_key_version: "encryptionkey@0"
I20260928 23:49:49.531148 9143 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260928 23:49:49.533615 9158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:49.534785 9143 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 23:49:49.534922 9143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "ce422dbbe094457b990271ea6891b8e9"
format_stamp: "Formatted at 2026-09-28 23:49:49 on dist-test-slave-p5bw"
server_key: "f9f2258aaf820be627b9a4bb9effbe48"
server_key_iv: "55ff8119da0b9ab8893ef9463f0baeff"
server_key_version: "encryptionkey@0"
I20260928 23:49:49.535049 9143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:49.549017 9143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:49.552150 9143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:49.552366 9143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:49.559317 9143 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:39831
I20260928 23:49:49.559340 9210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:39831 every 8 connection(s)
I20260928 23:49:49.560267 9143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:49:49.563337 9211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:49.569329 9211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce422dbbe094457b990271ea6891b8e9: Bootstrap starting.
I20260928 23:49:49.570032 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 9143
I20260928 23:49:49.570188 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:49:49.570454 6752 external_mini_cluster.cc:1428] Setting key d3d80fa085a821cc0d938e91b4d59462
I20260928 23:49:49.572331 9211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce422dbbe094457b990271ea6891b8e9: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:49.573158 9211 log.cc:824] T 00000000000000000000000000000000 P ce422dbbe094457b990271ea6891b8e9: Log is configured to *not* fsync() on all Append() calls
I20260928 23:49:49.575286 9211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce422dbbe094457b990271ea6891b8e9: No bootstrap required, opened a new log
Sep 28 23:49:49 dist-test-slave-p5bw krb5kdc[9130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:49.579574 9211 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce422dbbe094457b990271ea6891b8e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce422dbbe094457b990271ea6891b8e9" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 39831 } }
I20260928 23:49:49.579799 9211 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce422dbbe094457b990271ea6891b8e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:49.579869 9211 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce422dbbe094457b990271ea6891b8e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce422dbbe094457b990271ea6891b8e9, State: Initialized, Role: FOLLOWER
I20260928 23:49:49.580403 9211 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce422dbbe094457b990271ea6891b8e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce422dbbe094457b990271ea6891b8e9" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 39831 } }
I20260928 23:49:49.580554 9211 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce422dbbe094457b990271ea6891b8e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:49:49.580641 9211 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce422dbbe094457b990271ea6891b8e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:49:49.580750 9211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce422dbbe094457b990271ea6891b8e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:49.582050 9211 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce422dbbe094457b990271ea6891b8e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce422dbbe094457b990271ea6891b8e9" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 39831 } }
I20260928 23:49:49.582451 9211 leader_election.cc:302] T 00000000000000000000000000000000 P ce422dbbe094457b990271ea6891b8e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce422dbbe094457b990271ea6891b8e9; no voters:
I20260928 23:49:49.582849 9211 leader_election.cc:288] T 00000000000000000000000000000000 P ce422dbbe094457b990271ea6891b8e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:49:49.582985 9216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce422dbbe094457b990271ea6891b8e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:49:49.583240 9216 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce422dbbe094457b990271ea6891b8e9 [term 1 LEADER]: Becoming Leader. State: Replica: ce422dbbe094457b990271ea6891b8e9, State: Running, Role: LEADER
I20260928 23:49:49.583572 9216 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce422dbbe094457b990271ea6891b8e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce422dbbe094457b990271ea6891b8e9" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 39831 } }
I20260928 23:49:49.584321 9211 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce422dbbe094457b990271ea6891b8e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:49:49.585217 9217 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce422dbbe094457b990271ea6891b8e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce422dbbe094457b990271ea6891b8e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce422dbbe094457b990271ea6891b8e9" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 39831 } } }
I20260928 23:49:49.585343 9217 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce422dbbe094457b990271ea6891b8e9 [sys.catalog]: This master's current role is: LEADER
I20260928 23:49:49.585719 9218 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce422dbbe094457b990271ea6891b8e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce422dbbe094457b990271ea6891b8e9. Latest consensus state: current_term: 1 leader_uuid: "ce422dbbe094457b990271ea6891b8e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce422dbbe094457b990271ea6891b8e9" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 39831 } } }
I20260928 23:49:49.585846 9218 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce422dbbe094457b990271ea6891b8e9 [sys.catalog]: This master's current role is: LEADER
I20260928 23:49:49.586608 9214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:49.571766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54900 (local address 127.6.152.62:39831)
0928 23:49:49.572216 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:49.572231 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:49.572266 (+ 35us) server_negotiation.cc:407] Connection header received
0928 23:49:49.573018 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:49.573037 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:49.573409 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:49.573736 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:49.574546 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:49.575593 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:49.576439 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:49.576970 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:49.577135 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:49.580050 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:49.580056 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:49.580073 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:49.580108 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:49.582380 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:49.582938 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:49.582945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:49.582951 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:49.583109 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:49.583473 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:49.583476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:49.583478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:49.583996 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:49.584198 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:49.585001 (+ 803us) server_negotiation.cc:299] Negotiation successful
0928 23:49:49.585343 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":143,"threads_started":1}
I20260928 23:49:49.587054 9225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:49:49.590516 9225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:49:49.596257 9225 catalog_manager.cc:1347] Generated new cluster ID: daa6c2e437974cceaae40cffdcf12dd9
I20260928 23:49:49.596354 9225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:49:49.605453 9225 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:49:49.606524 9225 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:49:49.613695 9225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce422dbbe094457b990271ea6891b8e9: Generated new TSK 0
I20260928 23:49:49.614445 9225 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
I20260928 23:49:49.643934 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:39831
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:49.752588 9237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:49.752877 9237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:49.752959 9237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:49.756901 9237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:49.756950 9237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:49.757047 9237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:49:49.762426 9237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:39831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:49.763463 9237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:49.764456 9237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:49.773662 9242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:49.773701 9243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:49.773705 9245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:49.773854 9237 server_base.cc:1034] running on GCE node
I20260928 23:49:49.774396 9237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:49.775027 9237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:49.776202 9237 hybrid_clock.cc:648] HybridClock initialized: now 1790639389776188 us; error 24 us; skew 500 ppm
Sep 28 23:49:49 dist-test-slave-p5bw krb5kdc[9130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639389, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:49.779431 9237 init.cc:377] Logged in from keytab as kudu/127.6.152.1@KRBTEST.COM (short username kudu)
I20260928 23:49:49.780267 9237 webserver.cc:460] Webserver started at http://127.6.152.1:39549/ using document root <none> and password file <none>
I20260928 23:49:49.780761 9237 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:49.780869 9237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:49.781066 9237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:49.782974 9237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "cc0a0440dfc846f190243c5ae54f0743"
format_stamp: "Formatted at 2026-09-28 23:49:49 on dist-test-slave-p5bw"
server_key: "65823cf123e645ee06eb0522c67f664e"
server_key_iv: "4dd9facc18daf97467f58b060ecda58b"
server_key_version: "encryptionkey@0"
I20260928 23:49:49.783468 9237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "cc0a0440dfc846f190243c5ae54f0743"
format_stamp: "Formatted at 2026-09-28 23:49:49 on dist-test-slave-p5bw"
server_key: "65823cf123e645ee06eb0522c67f664e"
server_key_iv: "4dd9facc18daf97467f58b060ecda58b"
server_key_version: "encryptionkey@0"
I20260928 23:49:49.787293 9237 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:49:49.789592 9252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:49.790766 9237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:49:49.790884 9237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "cc0a0440dfc846f190243c5ae54f0743"
format_stamp: "Formatted at 2026-09-28 23:49:49 on dist-test-slave-p5bw"
server_key: "65823cf123e645ee06eb0522c67f664e"
server_key_iv: "4dd9facc18daf97467f58b060ecda58b"
server_key_version: "encryptionkey@0"
I20260928 23:49:49.790982 9237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:49.804149 9237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:49.806913 9237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:49.807112 9237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:49.807668 9237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:49:49.808444 9237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:49.808488 9237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:49.808552 9237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:49.808584 9237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:49.817618 9237 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:40269
I20260928 23:49:49.817639 9365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:40269 every 8 connection(s)
I20260928 23:49:49.818486 9237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:49:49.820465 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 9237
I20260928 23:49:49.820596 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
I20260928 23:49:49.820920 6752 external_mini_cluster.cc:1428] Setting key 4fa816db09cc6fc42cc12f08ec554c64
Sep 28 23:49:49 dist-test-slave-p5bw krb5kdc[9130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639389, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.152.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.2@KRBTEST.COM" created.
I20260928 23:49:49.832901 9214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:49.820187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:34043 (local address 127.6.152.62:39831)
0928 23:49:49.820356 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:49.820361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:49.821299 (+ 938us) server_negotiation.cc:407] Connection header received
0928 23:49:49.822890 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:49.822893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:49.822952 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:49.823017 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:49.824392 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:49.825044 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:49.826189 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:49.826428 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:49.826539 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:49.829519 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:49.829522 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:49.829524 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:49.829550 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:49.830968 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:49.831437 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:49.831439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:49.831440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:49.831513 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:49.831818 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:49.831824 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:49.831829 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:49.832034 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:49.832111 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:49.832599 (+ 488us) server_negotiation.cc:299] Negotiation successful
0928 23:49:49.832722 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 23:49:49.833730 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:49.820614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:39831 (local address 127.6.152.1:34043)
0928 23:49:49.821128 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:49.821181 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:49.822682 (+ 1501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:49.823176 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:49.823182 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:49.823519 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:49.824235 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:49.824257 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:49.825160 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:49.825163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:49.826053 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:49.826062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:49.826573 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:49.826578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:49.826759 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:49.827346 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:49.827409 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:49.829322 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:49.831094 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:49.831099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:49.831110 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:49.831351 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:49.831594 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:49.831597 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:49.831598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:49.831721 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:49.832120 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:49.832125 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:49.832338 (+ 213us) client_negotiation.cc:769] Sending connection context
0928 23:49:49.832536 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 23:49:49.832909 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":138,"threads_started":1}
I20260928 23:49:49.835212 9366 heartbeater.cc:347] Connected to a master server at 127.6.152.62:39831
I20260928 23:49:49.835481 9366 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:49.836023 9366 heartbeater.cc:511] Master 127.6.152.62:39831 requested a full tablet report, sending...
I20260928 23:49:49.837594 9175 ts_manager.cc:194] Registered new tserver with Master: cc0a0440dfc846f190243c5ae54f0743 (127.6.152.1:40269)
I20260928 23:49:49.838828 9175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.1@KRBTEST.COM'} at 127.6.152.1:34043
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
I20260928 23:49:49.848194 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:39831
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:49.961875 9371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:49.962170 9371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:49.962251 9371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:49.966370 9371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:49.966463 9371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:49.966559 9371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:49:49.972016 9371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:39831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:49.973088 9371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:49.974083 9371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:49.983636 9376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:49.983762 9379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:49.983754 9377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:49.983834 9371 server_base.cc:1034] running on GCE node
I20260928 23:49:49.984320 9371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:49.985116 9371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:49.986301 9371 hybrid_clock.cc:648] HybridClock initialized: now 1790639389986283 us; error 33 us; skew 500 ppm
Sep 28 23:49:49 dist-test-slave-p5bw krb5kdc[9130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639389, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:49.989158 9371 init.cc:377] Logged in from keytab as kudu/127.6.152.2@KRBTEST.COM (short username kudu)
I20260928 23:49:49.990024 9371 webserver.cc:460] Webserver started at http://127.6.152.2:36551/ using document root <none> and password file <none>
I20260928 23:49:49.990597 9371 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:49.990690 9371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:49.990929 9371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:49.993052 9371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "f1f9a3054a7f449dadf226d07ba406c6"
format_stamp: "Formatted at 2026-09-28 23:49:49 on dist-test-slave-p5bw"
server_key: "ae546211e3e4f7ac800801f40ebe6356"
server_key_iv: "d825a4fe698644229e9653a7d19db107"
server_key_version: "encryptionkey@0"
I20260928 23:49:49.993701 9371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "f1f9a3054a7f449dadf226d07ba406c6"
format_stamp: "Formatted at 2026-09-28 23:49:49 on dist-test-slave-p5bw"
server_key: "ae546211e3e4f7ac800801f40ebe6356"
server_key_iv: "d825a4fe698644229e9653a7d19db107"
server_key_version: "encryptionkey@0"
I20260928 23:49:49.997468 9371 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 23:49:49.999917 9386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:50.001240 9371 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:49:50.001374 9371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "f1f9a3054a7f449dadf226d07ba406c6"
format_stamp: "Formatted at 2026-09-28 23:49:49 on dist-test-slave-p5bw"
server_key: "ae546211e3e4f7ac800801f40ebe6356"
server_key_iv: "d825a4fe698644229e9653a7d19db107"
server_key_version: "encryptionkey@0"
I20260928 23:49:50.001487 9371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:50.022176 9371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:50.025457 9371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:50.025660 9371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:50.026258 9371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:49:50.027072 9371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:50.027144 9371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:50.027215 9371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:50.027258 9371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:50.036593 9371 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:39673
I20260928 23:49:50.036612 9499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:39673 every 8 connection(s)
I20260928 23:49:50.037552 9371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:49:50.044502 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 9371
I20260928 23:49:50.044643 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:49:50.045008 6752 external_mini_cluster.cc:1428] Setting key 847e483bc9cedd86aa222bde2494497c
Sep 28 23:49:50 dist-test-slave-p5bw krb5kdc[9130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639389, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:50.052721 9214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:50.039399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:56103 (local address 127.6.152.62:39831)
0928 23:49:50.039583 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:50.039587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:50.040468 (+ 881us) server_negotiation.cc:407] Connection header received
0928 23:49:50.041987 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:50.041989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:50.042035 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:50.042115 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:50.043401 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:50.043954 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:50.045502 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:50.045803 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:50.045917 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:50.048713 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:50.048716 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:50.048718 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:50.048743 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:50.050386 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:50.051039 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:50.051042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:50.051043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:50.051120 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:50.051538 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:50.051541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:50.051542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:50.051733 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:50.051806 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:50.052489 (+ 683us) server_negotiation.cc:299] Negotiation successful
0928 23:49:50.052601 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 23:49:50.053990 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:50.039861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:39831 (local address 127.6.152.2:56103)
0928 23:49:50.040304 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:50.040354 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:50.041774 (+ 1420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:50.042287 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:50.042294 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:50.042688 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:50.043248 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:50.043262 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:50.044078 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:50.044081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:50.045344 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:50.045355 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:50.045949 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:50.045953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:50.046099 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:50.046791 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:50.046807 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:50.048533 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:50.050506 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:50.050513 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:50.050552 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:50.050934 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:50.051215 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:50.051220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:50.051224 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:50.051432 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:50.051834 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:50.051844 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:50.052195 (+ 351us) client_negotiation.cc:769] Sending connection context
0928 23:49:50.052434 (+ 239us) client_negotiation.cc:240] Negotiation successful
0928 23:49:50.052801 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.6.152.3@KRBTEST.COM; defaulting to no policy
I20260928 23:49:50.055224 9500 heartbeater.cc:347] Connected to a master server at 127.6.152.62:39831
I20260928 23:49:50.055500 9500 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:50.056043 9500 heartbeater.cc:511] Master 127.6.152.62:39831 requested a full tablet report, sending...
I20260928 23:49:50.057118 9175 ts_manager.cc:194] Registered new tserver with Master: f1f9a3054a7f449dadf226d07ba406c6 (127.6.152.2:39673)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.3@KRBTEST.COM" created.
I20260928 23:49:50.057753 9175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.2@KRBTEST.COM'} at 127.6.152.2:56103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
I20260928 23:49:50.074288 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:39831
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:50.189291 9505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:50.189531 9505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:50.189610 9505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:50.193678 9505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:50.193732 9505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:50.193835 9505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:49:50.199023 9505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:39831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:50.199976 9505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:50.200978 9505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:50.210620 9511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:50.210745 9505 server_base.cc:1034] running on GCE node
W20260928 23:49:50.210619 9513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:50.210632 9510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:50.211225 9505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:50.211863 9505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:50.213044 9505 hybrid_clock.cc:648] HybridClock initialized: now 1790639390213031 us; error 23 us; skew 500 ppm
Sep 28 23:49:50 dist-test-slave-p5bw krb5kdc[9130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639390, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:50.215677 9505 init.cc:377] Logged in from keytab as kudu/127.6.152.3@KRBTEST.COM (short username kudu)
I20260928 23:49:50.216476 9505 webserver.cc:460] Webserver started at http://127.6.152.3:42381/ using document root <none> and password file <none>
I20260928 23:49:50.217085 9505 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:50.217173 9505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:50.217406 9505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:50.219393 9505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "88767d0f7f014720a8ececf600d6f688"
format_stamp: "Formatted at 2026-09-28 23:49:50 on dist-test-slave-p5bw"
server_key: "545ec9f45c1b164a0aa0cae31de8f550"
server_key_iv: "c1e01f948b47afa28d3a3686486e75bd"
server_key_version: "encryptionkey@0"
I20260928 23:49:50.219966 9505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "88767d0f7f014720a8ececf600d6f688"
format_stamp: "Formatted at 2026-09-28 23:49:50 on dist-test-slave-p5bw"
server_key: "545ec9f45c1b164a0aa0cae31de8f550"
server_key_iv: "c1e01f948b47afa28d3a3686486e75bd"
server_key_version: "encryptionkey@0"
I20260928 23:49:50.223453 9505 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 23:49:50.225832 9520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:50.227036 9505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:49:50.227171 9505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "88767d0f7f014720a8ececf600d6f688"
format_stamp: "Formatted at 2026-09-28 23:49:50 on dist-test-slave-p5bw"
server_key: "545ec9f45c1b164a0aa0cae31de8f550"
server_key_iv: "c1e01f948b47afa28d3a3686486e75bd"
server_key_version: "encryptionkey@0"
I20260928 23:49:50.227284 9505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:50.239688 9505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:50.243175 9505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:50.243395 9505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:50.243996 9505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:49:50.244853 9505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:50.244901 9505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:50.244975 9505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:50.245020 9505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:50.254804 9505 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:35767
I20260928 23:49:50.254827 9633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:35767 every 8 connection(s)
I20260928 23:49:50.255779 9505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:49:50.260489 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 9505
I20260928 23:49:50.260577 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:49:50.260932 6752 external_mini_cluster.cc:1428] Setting key 7e74e3de76313c60208ae0c937c2df7a
Sep 28 23:49:50 dist-test-slave-p5bw krb5kdc[9130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639390, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:50.269546 9214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:50.257632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:34659 (local address 127.6.152.62:39831)
0928 23:49:50.257766 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:50.257769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:50.258548 (+ 779us) server_negotiation.cc:407] Connection header received
0928 23:49:50.259940 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:50.259943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:50.260004 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:50.260106 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:50.261607 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:50.262322 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:50.263620 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:50.263857 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:50.263965 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:50.266312 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:50.266315 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:50.266318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:50.266345 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:50.267570 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:50.268077 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:50.268081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:50.268083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:50.268164 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:50.268505 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:50.268509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:50.268511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:50.268683 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:50.268791 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:50.269276 (+ 485us) server_negotiation.cc:299] Negotiation successful
0928 23:49:50.269389 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:49:50.270283 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:50.257971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:39831 (local address 127.6.152.3:34659)
0928 23:49:50.258405 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:50.258446 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:50.259691 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:50.260288 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:50.260295 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:50.260748 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:50.261453 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:50.261479 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:50.262428 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:50.262432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:50.263483 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:50.263493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:50.264001 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:50.264005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:50.264182 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:50.264689 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:50.264704 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:50.266155 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:50.267685 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:50.267691 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:50.267704 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:50.267971 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:50.268261 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:50.268264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:50.268266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:50.268408 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:50.268787 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:50.268793 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:50.269048 (+ 255us) client_negotiation.cc:769] Sending connection context
0928 23:49:50.269237 (+ 189us) client_negotiation.cc:240] Negotiation successful
0928 23:49:50.269541 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":105,"threads_started":1}
I20260928 23:49:50.271335 9634 heartbeater.cc:347] Connected to a master server at 127.6.152.62:39831
I20260928 23:49:50.271535 9634 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:50.271961 9634 heartbeater.cc:511] Master 127.6.152.62:39831 requested a full tablet report, sending...
I20260928 23:49:50.272935 9175 ts_manager.cc:194] Registered new tserver with Master: 88767d0f7f014720a8ececf600d6f688 (127.6.152.3:35767)
I20260928 23:49:50.273440 9175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.3@KRBTEST.COM'} at 127.6.152.3:34659
I20260928 23:49:50.274643 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 23:49:50.287688 9214 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:49:50.286247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54908 (local address 127.6.152.62:39831)
0928 23:49:50.286381 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:50.286384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:50.286586 (+ 202us) server_negotiation.cc:407] Connection header received
0928 23:49:50.286706 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:50.286708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:50.286752 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:50.286822 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:50.287519 (+ 697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54908: BlockingRecv error: recv got EOF from 127.0.0.1:54908 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:49:50.289032 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 9237
I20260928 23:49:50.294816 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 9371
I20260928 23:49:50.300068 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 9505
I20260928 23:49:50.305339 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 9143
2026-09-28T23:49:50Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2075 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:49:50 dist-test-slave-p5bw krb5kdc[9646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:49:50 dist-test-slave-p5bw krb5kdc[9646](info): set up 2 sockets
Sep 28 23:49:50 dist-test-slave-p5bw krb5kdc[9646](info): commencing operation
krb5kdc: starting...
W20260928 23:49:51.381364 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:49:51 dist-test-slave-p5bw krb5kdc[9646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:49:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:49:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:49:51.500013 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:37613
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:39059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:37613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:51.607600 9659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:51.607916 9659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:51.608003 9659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:51.612007 9659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:49:51.612093 9659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:51.612133 9659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:51.612170 9659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:49:51.612200 9659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:49:51.617372 9659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:39059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:37613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:37613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:51.618396 9659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:51.619449 9659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:51.627943 9667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:51.627966 9664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:51.627943 9665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:51.628212 9659 server_base.cc:1034] running on GCE node
I20260928 23:49:51.628875 9659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:51.629968 9659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:51.631156 9659 hybrid_clock.cc:648] HybridClock initialized: now 1790639391631131 us; error 39 us; skew 500 ppm
Sep 28 23:49:51 dist-test-slave-p5bw krb5kdc[9646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639391, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:51.637143 9659 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:49:51.638067 9659 webserver.cc:460] Webserver started at http://127.6.152.62:38715/ using document root <none> and password file <none>
I20260928 23:49:51.638633 9659 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:51.638725 9659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:51.638964 9659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:51.641057 9659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "12f5b2be3ab042a0bfbe77ff372bd533"
format_stamp: "Formatted at 2026-09-28 23:49:51 on dist-test-slave-p5bw"
server_key: "1e9d9733a6821027055708027223a124"
server_key_iv: "ac856b39a35386ca6fcd494cee02e5dc"
server_key_version: "encryptionkey@0"
I20260928 23:49:51.641610 9659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "12f5b2be3ab042a0bfbe77ff372bd533"
format_stamp: "Formatted at 2026-09-28 23:49:51 on dist-test-slave-p5bw"
server_key: "1e9d9733a6821027055708027223a124"
server_key_iv: "ac856b39a35386ca6fcd494cee02e5dc"
server_key_version: "encryptionkey@0"
I20260928 23:49:51.645339 9659 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:49:51.647734 9674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:51.648976 9659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:49:51.649111 9659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "12f5b2be3ab042a0bfbe77ff372bd533"
format_stamp: "Formatted at 2026-09-28 23:49:51 on dist-test-slave-p5bw"
server_key: "1e9d9733a6821027055708027223a124"
server_key_iv: "ac856b39a35386ca6fcd494cee02e5dc"
server_key_version: "encryptionkey@0"
I20260928 23:49:51.649222 9659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:51.677277 9659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:51.680214 9659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:51.680426 9659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:51.687328 9659 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:37613
I20260928 23:49:51.687623 9726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:37613 every 8 connection(s)
I20260928 23:49:51.688315 9659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:49:51.691061 9727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:51.695855 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 9659
I20260928 23:49:51.695998 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:49:51.696260 6752 external_mini_cluster.cc:1428] Setting key 34b7bd198ca83a0d2f7d222858098b0e
I20260928 23:49:51.696445 9727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f5b2be3ab042a0bfbe77ff372bd533: Bootstrap starting.
I20260928 23:49:51.698882 9727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12f5b2be3ab042a0bfbe77ff372bd533: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:51.699837 9727 log.cc:824] T 00000000000000000000000000000000 P 12f5b2be3ab042a0bfbe77ff372bd533: Log is configured to *not* fsync() on all Append() calls
I20260928 23:49:51.702036 9727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f5b2be3ab042a0bfbe77ff372bd533: No bootstrap required, opened a new log
Sep 28 23:49:51 dist-test-slave-p5bw krb5kdc[9646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:51.706498 9727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12f5b2be3ab042a0bfbe77ff372bd533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f5b2be3ab042a0bfbe77ff372bd533" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 37613 } }
I20260928 23:49:51.706804 9727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12f5b2be3ab042a0bfbe77ff372bd533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:51.706857 9727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12f5b2be3ab042a0bfbe77ff372bd533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f5b2be3ab042a0bfbe77ff372bd533, State: Initialized, Role: FOLLOWER
I20260928 23:49:51.707386 9727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12f5b2be3ab042a0bfbe77ff372bd533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f5b2be3ab042a0bfbe77ff372bd533" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 37613 } }
I20260928 23:49:51.707540 9727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12f5b2be3ab042a0bfbe77ff372bd533 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:49:51.707610 9727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12f5b2be3ab042a0bfbe77ff372bd533 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:49:51.707713 9727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12f5b2be3ab042a0bfbe77ff372bd533 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:51.708833 9727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12f5b2be3ab042a0bfbe77ff372bd533 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f5b2be3ab042a0bfbe77ff372bd533" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 37613 } }
I20260928 23:49:51.709188 9727 leader_election.cc:302] T 00000000000000000000000000000000 P 12f5b2be3ab042a0bfbe77ff372bd533 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12f5b2be3ab042a0bfbe77ff372bd533; no voters:
I20260928 23:49:51.709517 9727 leader_election.cc:288] T 00000000000000000000000000000000 P 12f5b2be3ab042a0bfbe77ff372bd533 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:49:51.709620 9732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12f5b2be3ab042a0bfbe77ff372bd533 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:49:51.709878 9732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12f5b2be3ab042a0bfbe77ff372bd533 [term 1 LEADER]: Becoming Leader. State: Replica: 12f5b2be3ab042a0bfbe77ff372bd533, State: Running, Role: LEADER
I20260928 23:49:51.710250 9732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12f5b2be3ab042a0bfbe77ff372bd533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f5b2be3ab042a0bfbe77ff372bd533" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 37613 } }
I20260928 23:49:51.710783 9727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12f5b2be3ab042a0bfbe77ff372bd533 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:49:51.711082 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:51.697511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34520 (local address 127.6.152.62:37613)
0928 23:49:51.697909 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:51.697925 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:51.697965 (+ 40us) server_negotiation.cc:407] Connection header received
0928 23:49:51.698571 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:51.698586 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:51.698909 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:51.699251 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:51.700162 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:51.701380 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:51.702532 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:51.702998 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:51.703113 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:51.705919 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:51.705927 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:51.705943 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:51.705975 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:51.707902 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:51.708313 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:51.708319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:51.708325 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:51.708483 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:51.708770 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:51.708774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:51.708776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:51.709318 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:51.709533 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:51.709861 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 23:49:51.710149 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":125,"threads_started":1}
I20260928 23:49:51.712270 9733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f5b2be3ab042a0bfbe77ff372bd533 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12f5b2be3ab042a0bfbe77ff372bd533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f5b2be3ab042a0bfbe77ff372bd533" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 37613 } } }
I20260928 23:49:51.712381 9733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f5b2be3ab042a0bfbe77ff372bd533 [sys.catalog]: This master's current role is: LEADER
I20260928 23:49:51.712585 9733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f5b2be3ab042a0bfbe77ff372bd533 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12f5b2be3ab042a0bfbe77ff372bd533. Latest consensus state: current_term: 1 leader_uuid: "12f5b2be3ab042a0bfbe77ff372bd533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f5b2be3ab042a0bfbe77ff372bd533" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 37613 } } }
I20260928 23:49:51.712652 9733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f5b2be3ab042a0bfbe77ff372bd533 [sys.catalog]: This master's current role is: LEADER
I20260928 23:49:51.713254 9746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:49:51.714387 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:49:51.716313 9746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:49:51.721690 9746 catalog_manager.cc:1347] Generated new cluster ID: c4216dc96e904d7ca64b640e3b71cabe
I20260928 23:49:51.721787 9746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:49:51.731794 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:51.727094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34526 (local address 127.6.152.62:37613)
0928 23:49:51.727254 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:51.727259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:51.727522 (+ 263us) server_negotiation.cc:407] Connection header received
0928 23:49:51.727647 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:51.727650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:51.727702 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:51.727789 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:51.728310 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:51.728866 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:51.729702 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:51.729964 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:51.730077 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:51.730914 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:51.730917 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:51.730920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:49:51.731368 (+ 448us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:51.731394 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:49:51.731410 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:51.731517 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:51.731599 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 23:49:51.731654 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 23:49:51.733665 9746 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:49:51.734520 9746 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:49:51.740324 9746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12f5b2be3ab042a0bfbe77ff372bd533: Generated new TSK 0
I20260928 23:49:51.741169 9746 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:49:51.746246 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 9659
2026-09-28T23:49:51Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1438 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:49:51 dist-test-slave-p5bw krb5kdc[9760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:49:51 dist-test-slave-p5bw krb5kdc[9760](info): set up 2 sockets
Sep 28 23:49:51 dist-test-slave-p5bw krb5kdc[9760](info): commencing operation
krb5kdc: starting...
W20260928 23:49:52.781446 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:49:52 dist-test-slave-p5bw krb5kdc[9760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:49:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:49:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:49:52.903246 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:37545
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:37545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:53.011229 9773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:53.011507 9773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:53.011548 9773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:53.015437 9773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:49:53.015496 9773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:53.015515 9773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:53.015532 9773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:49:53.015547 9773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:49:53.020658 9773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:37545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:37545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:53.021704 9773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:53.022632 9773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:53.030983 9781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:53.030965 9779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:53.030949 9778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:53.031073 9773 server_base.cc:1034] running on GCE node
I20260928 23:49:53.032053 9773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:53.033218 9773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:53.034410 9773 hybrid_clock.cc:648] HybridClock initialized: now 1790639393034403 us; error 36 us; skew 500 ppm
Sep 28 23:49:53 dist-test-slave-p5bw krb5kdc[9760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639393, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:53.037096 9773 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:49:53.037897 9773 webserver.cc:460] Webserver started at http://127.6.152.62:40359/ using document root <none> and password file <none>
I20260928 23:49:53.038410 9773 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:53.038466 9773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:53.038622 9773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:53.040565 9773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "c463f985dd01497eb16ca03813f6da30"
format_stamp: "Formatted at 2026-09-28 23:49:53 on dist-test-slave-p5bw"
server_key: "a0decef06840d2bcbc575c2d8863c8aa"
server_key_iv: "347ea459eab812a25d9a4e25e7cbe970"
server_key_version: "encryptionkey@0"
I20260928 23:49:53.041131 9773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "c463f985dd01497eb16ca03813f6da30"
format_stamp: "Formatted at 2026-09-28 23:49:53 on dist-test-slave-p5bw"
server_key: "a0decef06840d2bcbc575c2d8863c8aa"
server_key_iv: "347ea459eab812a25d9a4e25e7cbe970"
server_key_version: "encryptionkey@0"
I20260928 23:49:53.044654 9773 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 23:49:53.047019 9788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:53.048244 9773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:49:53.048352 9773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "c463f985dd01497eb16ca03813f6da30"
format_stamp: "Formatted at 2026-09-28 23:49:53 on dist-test-slave-p5bw"
server_key: "a0decef06840d2bcbc575c2d8863c8aa"
server_key_iv: "347ea459eab812a25d9a4e25e7cbe970"
server_key_version: "encryptionkey@0"
I20260928 23:49:53.048434 9773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:53.074951 9773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:53.078040 9773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:53.078197 9773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:53.085413 9773 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:37545
I20260928 23:49:53.085443 9840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:37545 every 8 connection(s)
I20260928 23:49:53.086370 9773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:49:53.089175 9841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:53.089365 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 9773
I20260928 23:49:53.089510 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:49:53.089802 6752 external_mini_cluster.cc:1428] Setting key 8af4e4da426af896967d7607a249e280
I20260928 23:49:53.095801 9841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c463f985dd01497eb16ca03813f6da30: Bootstrap starting.
I20260928 23:49:53.098515 9841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c463f985dd01497eb16ca03813f6da30: Neither blocks nor log segments found. Creating new log.
Sep 28 23:49:53 dist-test-slave-p5bw krb5kdc[9760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:53.099260 9841 log.cc:824] T 00000000000000000000000000000000 P c463f985dd01497eb16ca03813f6da30: Log is configured to *not* fsync() on all Append() calls
I20260928 23:49:53.101536 9841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c463f985dd01497eb16ca03813f6da30: No bootstrap required, opened a new log
I20260928 23:49:53.104506 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:53.091271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53446 (local address 127.6.152.62:37545)
0928 23:49:53.091714 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:53.091730 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:53.091769 (+ 39us) server_negotiation.cc:407] Connection header received
0928 23:49:53.092475 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:53.092492 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:53.092893 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:53.093196 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:53.094108 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:53.095180 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:53.096314 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:53.096863 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:53.097041 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:53.099881 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:53.099889 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:53.099907 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:53.099942 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:53.101738 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:53.102244 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:53.102249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:53.102253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:53.102358 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:53.102666 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:53.102669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:53.102670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:53.103046 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:53.103179 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:53.103507 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 23:49:53.103800 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":140,"threads_started":1}
I20260928 23:49:53.106141 9841 raft_consensus.cc:348] T 00000000000000000000000000000000 P c463f985dd01497eb16ca03813f6da30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c463f985dd01497eb16ca03813f6da30" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 37545 } }
I20260928 23:49:53.106431 9841 raft_consensus.cc:374] T 00000000000000000000000000000000 P c463f985dd01497eb16ca03813f6da30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:53.106487 9841 raft_consensus.cc:729] T 00000000000000000000000000000000 P c463f985dd01497eb16ca03813f6da30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c463f985dd01497eb16ca03813f6da30, State: Initialized, Role: FOLLOWER
I20260928 23:49:53.106968 9841 consensus_queue.cc:260] T 00000000000000000000000000000000 P c463f985dd01497eb16ca03813f6da30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c463f985dd01497eb16ca03813f6da30" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 37545 } }
I20260928 23:49:53.107151 9841 raft_consensus.cc:388] T 00000000000000000000000000000000 P c463f985dd01497eb16ca03813f6da30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:49:53.107206 9841 raft_consensus.cc:482] T 00000000000000000000000000000000 P c463f985dd01497eb16ca03813f6da30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:49:53.107283 9841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c463f985dd01497eb16ca03813f6da30 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:53.108273 9841 raft_consensus.cc:504] T 00000000000000000000000000000000 P c463f985dd01497eb16ca03813f6da30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c463f985dd01497eb16ca03813f6da30" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 37545 } }
I20260928 23:49:53.108592 9841 leader_election.cc:302] T 00000000000000000000000000000000 P c463f985dd01497eb16ca03813f6da30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c463f985dd01497eb16ca03813f6da30; no voters:
I20260928 23:49:53.108922 9841 leader_election.cc:288] T 00000000000000000000000000000000 P c463f985dd01497eb16ca03813f6da30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:49:53.109084 9846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c463f985dd01497eb16ca03813f6da30 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:49:53.109324 9846 raft_consensus.cc:686] T 00000000000000000000000000000000 P c463f985dd01497eb16ca03813f6da30 [term 1 LEADER]: Becoming Leader. State: Replica: c463f985dd01497eb16ca03813f6da30, State: Running, Role: LEADER
I20260928 23:49:53.109645 9846 consensus_queue.cc:237] T 00000000000000000000000000000000 P c463f985dd01497eb16ca03813f6da30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c463f985dd01497eb16ca03813f6da30" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 37545 } }
I20260928 23:49:53.110299 9841 sys_catalog.cc:564] T 00000000000000000000000000000000 P c463f985dd01497eb16ca03813f6da30 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:49:53.111356 9848 sys_catalog.cc:455] T 00000000000000000000000000000000 P c463f985dd01497eb16ca03813f6da30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c463f985dd01497eb16ca03813f6da30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c463f985dd01497eb16ca03813f6da30" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 37545 } } }
I20260928 23:49:53.111481 9848 sys_catalog.cc:458] T 00000000000000000000000000000000 P c463f985dd01497eb16ca03813f6da30 [sys.catalog]: This master's current role is: LEADER
I20260928 23:49:53.111881 9855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:49:53.111852 9847 sys_catalog.cc:455] T 00000000000000000000000000000000 P c463f985dd01497eb16ca03813f6da30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c463f985dd01497eb16ca03813f6da30. Latest consensus state: current_term: 1 leader_uuid: "c463f985dd01497eb16ca03813f6da30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c463f985dd01497eb16ca03813f6da30" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 37545 } } }
I20260928 23:49:53.112030 9847 sys_catalog.cc:458] T 00000000000000000000000000000000 P c463f985dd01497eb16ca03813f6da30 [sys.catalog]: This master's current role is: LEADER
I20260928 23:49:53.114813 9855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:49:53.121093 9855 catalog_manager.cc:1347] Generated new cluster ID: 97eaadb539404871bf85d67604650a98
I20260928 23:49:53.121173 9855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:49:53.127022 9855 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:49:53.127904 9855 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:49:53.133148 9855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c463f985dd01497eb16ca03813f6da30: Generated new TSK 0
I20260928 23:49:53.133935 9855 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
I20260928 23:49:53.164189 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:37545
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:53.273571 9867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:53.273823 9867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:53.273865 9867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:53.277760 9867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:53.277809 9867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:53.277866 9867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:49:53.283030 9867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:37545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:53.284019 9867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:53.284983 9867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:53.294404 9875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:53.294384 9872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:53.294415 9873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:53.294790 9867 server_base.cc:1034] running on GCE node
I20260928 23:49:53.295223 9867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:53.295783 9867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:53.297046 9867 hybrid_clock.cc:648] HybridClock initialized: now 1790639393297025 us; error 32 us; skew 500 ppm
Sep 28 23:49:53 dist-test-slave-p5bw krb5kdc[9760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639393, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:53.299943 9867 init.cc:377] Logged in from keytab as kudu/127.6.152.1@KRBTEST.COM (short username kudu)
I20260928 23:49:53.300870 9867 webserver.cc:460] Webserver started at http://127.6.152.1:44517/ using document root <none> and password file <none>
I20260928 23:49:53.301446 9867 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:53.301535 9867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:53.301766 9867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:53.303773 9867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "923eb9f7ceb44ceaad04f8f294605b9f"
format_stamp: "Formatted at 2026-09-28 23:49:53 on dist-test-slave-p5bw"
server_key: "2487c7a7d625ac32bb2bf5c3018e1962"
server_key_iv: "ab9d1ae2e76482da629fa2d873fd60f8"
server_key_version: "encryptionkey@0"
I20260928 23:49:53.304338 9867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "923eb9f7ceb44ceaad04f8f294605b9f"
format_stamp: "Formatted at 2026-09-28 23:49:53 on dist-test-slave-p5bw"
server_key: "2487c7a7d625ac32bb2bf5c3018e1962"
server_key_iv: "ab9d1ae2e76482da629fa2d873fd60f8"
server_key_version: "encryptionkey@0"
I20260928 23:49:53.308038 9867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 23:49:53.310495 9882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:53.311888 9867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:49:53.312006 9867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "923eb9f7ceb44ceaad04f8f294605b9f"
format_stamp: "Formatted at 2026-09-28 23:49:53 on dist-test-slave-p5bw"
server_key: "2487c7a7d625ac32bb2bf5c3018e1962"
server_key_iv: "ab9d1ae2e76482da629fa2d873fd60f8"
server_key_version: "encryptionkey@0"
I20260928 23:49:53.312129 9867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:53.330125 9867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:53.333114 9867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:53.333313 9867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:53.333889 9867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:49:53.334722 9867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:53.334794 9867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:53.334867 9867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:53.334910 9867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:53.344069 9867 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:39489
I20260928 23:49:53.344090 9995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:39489 every 8 connection(s)
I20260928 23:49:53.344978 9867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:49:53.350296 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 9867
I20260928 23:49:53.350414 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
I20260928 23:49:53.350703 6752 external_mini_cluster.cc:1428] Setting key 0eaded8dfc0f86189101dfe92ba43348
Sep 28 23:49:53 dist-test-slave-p5bw krb5kdc[9760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639393, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:53.359346 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:53.346822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:48049 (local address 127.6.152.62:37545)
0928 23:49:53.346979 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:53.346982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:53.347785 (+ 803us) server_negotiation.cc:407] Connection header received
0928 23:49:53.349223 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:53.349226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:53.349274 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:53.349371 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:53.350614 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:53.351259 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:53.352504 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:53.352749 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:53.352882 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:53.355500 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:53.355503 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:53.355505 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:53.355528 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:53.357175 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:53.357775 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:53.357778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:53.357780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:53.357867 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:53.358256 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:53.358261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:53.358263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:53.358452 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:53.358565 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:53.359054 (+ 489us) server_negotiation.cc:299] Negotiation successful
0928 23:49:53.359175 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 23:49:53.360138 9998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:53.347200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:37545 (local address 127.6.152.1:48049)
0928 23:49:53.347627 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:53.347669 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:53.348998 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:53.349509 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:53.349515 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:53.349859 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:53.350360 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:53.350404 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:53.351398 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:53.351402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:53.352379 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:53.352390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:53.352919 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:53.352932 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:53.353128 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:53.353754 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:53.353770 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:53.355316 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:53.357280 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:53.357286 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:53.357298 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:53.357658 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:53.358001 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:53.358004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:53.358006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:53.358171 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:53.358562 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:53.358570 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:53.358806 (+ 236us) client_negotiation.cc:769] Sending connection context
0928 23:49:53.358999 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 23:49:53.359316 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.6.152.2@KRBTEST.COM; defaulting to no policy
I20260928 23:49:53.361423 9996 heartbeater.cc:347] Connected to a master server at 127.6.152.62:37545
I20260928 23:49:53.361739 9996 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:53.362233 9996 heartbeater.cc:511] Master 127.6.152.62:37545 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.2@KRBTEST.COM" created.
I20260928 23:49:53.363875 9805 ts_manager.cc:194] Registered new tserver with Master: 923eb9f7ceb44ceaad04f8f294605b9f (127.6.152.1:39489)
I20260928 23:49:53.365200 9805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.1@KRBTEST.COM'} at 127.6.152.1:48049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
I20260928 23:49:53.381161 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:37545
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:53.489914 10001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:53.490162 10001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:53.490202 10001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:53.494134 10001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:53.494186 10001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:53.494277 10001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:49:53.499399 10001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:37545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:53.500371 10001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:53.501412 10001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:53.510906 10007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:53.511003 10006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:53.511021 10001 server_base.cc:1034] running on GCE node
W20260928 23:49:53.510918 10009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:53.511665 10001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:53.512292 10001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:53.513459 10001 hybrid_clock.cc:648] HybridClock initialized: now 1790639393513441 us; error 31 us; skew 500 ppm
Sep 28 23:49:53 dist-test-slave-p5bw krb5kdc[9760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639393, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:53.516274 10001 init.cc:377] Logged in from keytab as kudu/127.6.152.2@KRBTEST.COM (short username kudu)
I20260928 23:49:53.517204 10001 webserver.cc:460] Webserver started at http://127.6.152.2:36031/ using document root <none> and password file <none>
I20260928 23:49:53.517735 10001 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:53.517823 10001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:53.518050 10001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:53.520005 10001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "224c770335ca46c88e7e03825015e361"
format_stamp: "Formatted at 2026-09-28 23:49:53 on dist-test-slave-p5bw"
server_key: "67ac1e6c52d34339244b9f9662425177"
server_key_iv: "eecb31cd01d24fb4de08a9483871de90"
server_key_version: "encryptionkey@0"
I20260928 23:49:53.520538 10001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "224c770335ca46c88e7e03825015e361"
format_stamp: "Formatted at 2026-09-28 23:49:53 on dist-test-slave-p5bw"
server_key: "67ac1e6c52d34339244b9f9662425177"
server_key_iv: "eecb31cd01d24fb4de08a9483871de90"
server_key_version: "encryptionkey@0"
I20260928 23:49:53.524140 10001 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 23:49:53.526454 10016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:53.527650 10001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:49:53.527783 10001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "224c770335ca46c88e7e03825015e361"
format_stamp: "Formatted at 2026-09-28 23:49:53 on dist-test-slave-p5bw"
server_key: "67ac1e6c52d34339244b9f9662425177"
server_key_iv: "eecb31cd01d24fb4de08a9483871de90"
server_key_version: "encryptionkey@0"
I20260928 23:49:53.527921 10001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:53.545992 10001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:53.549247 10001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:53.549444 10001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:53.550012 10001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:49:53.550802 10001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:53.550877 10001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:53.550962 10001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:53.551012 10001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:53.559779 10001 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:40241
I20260928 23:49:53.559800 10129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:40241 every 8 connection(s)
I20260928 23:49:53.560655 10001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:49:53.567201 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 10001
I20260928 23:49:53.567342 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:49:53.567610 6752 external_mini_cluster.cc:1428] Setting key 4d86344678f969130e61b5bc48687b5d
Sep 28 23:49:53 dist-test-slave-p5bw krb5kdc[9760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639393, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:53.574992 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:53.562464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:37813 (local address 127.6.152.62:37545)
0928 23:49:53.562621 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:53.562626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:53.563464 (+ 838us) server_negotiation.cc:407] Connection header received
0928 23:49:53.564731 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:53.564733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:53.564779 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:53.564910 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:53.566022 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:53.566543 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:53.567810 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:53.568110 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:53.568224 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:53.571006 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:53.571009 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:53.571011 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:53.571035 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:53.572559 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:53.573246 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:53.573249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:53.573250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:53.573325 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:53.573734 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:53.573736 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:53.573737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:53.573904 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:53.573984 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:53.574698 (+ 714us) server_negotiation.cc:299] Negotiation successful
0928 23:49:53.574856 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 23:49:53.576139 10132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:53.562891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:37545 (local address 127.6.152.2:37813)
0928 23:49:53.563318 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:53.563359 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:53.564541 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:53.565080 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:53.565087 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:53.565434 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:53.565895 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:53.565909 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:53.566644 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:53.566647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:53.567656 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:53.567665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:53.568267 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:53.568270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:53.568411 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:53.569041 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:53.569056 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:53.570804 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:53.572680 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:53.572690 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:53.572709 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:53.573143 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:53.573423 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:53.573426 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:53.573428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:53.573637 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:53.574018 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:53.574026 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:53.574378 (+ 352us) client_negotiation.cc:769] Sending connection context
0928 23:49:53.574631 (+ 253us) client_negotiation.cc:240] Negotiation successful
0928 23:49:53.575034 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.6.152.3@KRBTEST.COM; defaulting to no policy
I20260928 23:49:53.577435 10130 heartbeater.cc:347] Connected to a master server at 127.6.152.62:37545
I20260928 23:49:53.577704 10130 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:53.578157 10130 heartbeater.cc:511] Master 127.6.152.62:37545 requested a full tablet report, sending...
I20260928 23:49:53.579238 9805 ts_manager.cc:194] Registered new tserver with Master: 224c770335ca46c88e7e03825015e361 (127.6.152.2:40241)
I20260928 23:49:53.579833 9805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.2@KRBTEST.COM'} at 127.6.152.2:37813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
I20260928 23:49:53.596037 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:37545
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:53.705317 10135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:53.705547 10135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:53.705587 10135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:53.709390 10135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:53.709432 10135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:53.709487 10135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:49:53.714617 10135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:37545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:53.715582 10135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:53.716544 10135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:53.725658 10140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:53.725732 10141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:53.725833 10135 server_base.cc:1034] running on GCE node
W20260928 23:49:53.725708 10143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:53.726313 10135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:53.726938 10135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:53.728123 10135 hybrid_clock.cc:648] HybridClock initialized: now 1790639393728088 us; error 45 us; skew 500 ppm
Sep 28 23:49:53 dist-test-slave-p5bw krb5kdc[9760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639393, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:53.730912 10135 init.cc:377] Logged in from keytab as kudu/127.6.152.3@KRBTEST.COM (short username kudu)
I20260928 23:49:53.731854 10135 webserver.cc:460] Webserver started at http://127.6.152.3:44541/ using document root <none> and password file <none>
I20260928 23:49:53.732386 10135 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:53.732484 10135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:53.732709 10135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:53.734684 10135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "8c74bc4f2e2a4b3da5b6b7eb1c1371fb"
format_stamp: "Formatted at 2026-09-28 23:49:53 on dist-test-slave-p5bw"
server_key: "6af17eb89c7b5999ee012ebe2dfcc015"
server_key_iv: "41407554891d4e7b77e8cfde16c03637"
server_key_version: "encryptionkey@0"
I20260928 23:49:53.735247 10135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "8c74bc4f2e2a4b3da5b6b7eb1c1371fb"
format_stamp: "Formatted at 2026-09-28 23:49:53 on dist-test-slave-p5bw"
server_key: "6af17eb89c7b5999ee012ebe2dfcc015"
server_key_iv: "41407554891d4e7b77e8cfde16c03637"
server_key_version: "encryptionkey@0"
I20260928 23:49:53.738754 10135 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 23:49:53.741016 10150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:53.742213 10135 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:49:53.742338 10135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "8c74bc4f2e2a4b3da5b6b7eb1c1371fb"
format_stamp: "Formatted at 2026-09-28 23:49:53 on dist-test-slave-p5bw"
server_key: "6af17eb89c7b5999ee012ebe2dfcc015"
server_key_iv: "41407554891d4e7b77e8cfde16c03637"
server_key_version: "encryptionkey@0"
I20260928 23:49:53.742449 10135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:53.786456 10135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:53.789921 10135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:53.790138 10135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:53.790740 10135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:49:53.791558 10135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:53.791628 10135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:53.791692 10135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:53.791742 10135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:53.801640 10135 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:41197
I20260928 23:49:53.801664 10263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:41197 every 8 connection(s)
I20260928 23:49:53.802582 10135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
Sep 28 23:49:53 dist-test-slave-p5bw krb5kdc[9760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639393, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:53.812659 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 10135
I20260928 23:49:53.812773 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:49:53.813153 6752 external_mini_cluster.cc:1428] Setting key 40db5492b65173b3c42b049407d6ea3f
I20260928 23:49:53.816788 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:53.804350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:47305 (local address 127.6.152.62:37545)
0928 23:49:53.804506 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:53.804510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:53.805464 (+ 954us) server_negotiation.cc:407] Connection header received
0928 23:49:53.806769 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:53.806772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:53.806819 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:53.806904 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:53.808053 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:53.808600 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:53.809755 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:53.810135 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:53.810231 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:53.813003 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:53.813010 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:53.813015 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:53.813048 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:53.814638 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:53.815145 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:53.815149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:53.815151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:53.815289 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:53.815663 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:53.815667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:53.815668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:53.815852 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:53.815975 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:53.816495 (+ 520us) server_negotiation.cc:299] Negotiation successful
0928 23:49:53.816597 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 23:49:53.817525 10266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:53.804799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:37545 (local address 127.6.152.3:47305)
0928 23:49:53.805306 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:53.805350 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:53.806560 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:53.807076 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:53.807083 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:53.807449 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:53.807909 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:53.807921 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:53.808706 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:53.808709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:53.809650 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:53.809659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:53.810272 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:53.810275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:53.810412 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:53.811051 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:53.811075 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:53.812790 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:53.814768 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:53.814774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:53.814789 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:53.815037 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:53.815381 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:53.815384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:53.815386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:53.815536 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:53.815976 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:53.815983 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:53.816213 (+ 230us) client_negotiation.cc:769] Sending connection context
0928 23:49:53.816417 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 23:49:53.816673 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":111,"threads_started":1}
I20260928 23:49:53.818668 10264 heartbeater.cc:347] Connected to a master server at 127.6.152.62:37545
I20260928 23:49:53.818873 10264 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:53.819294 10264 heartbeater.cc:511] Master 127.6.152.62:37545 requested a full tablet report, sending...
I20260928 23:49:53.820199 9805 ts_manager.cc:194] Registered new tserver with Master: 8c74bc4f2e2a4b3da5b6b7eb1c1371fb (127.6.152.3:41197)
I20260928 23:49:53.820755 9805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.3@KRBTEST.COM'} at 127.6.152.3:47305
I20260928 23:49:53.826948 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 23:49:53 dist-test-slave-p5bw krb5kdc[9760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639393, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 23:49:53 dist-test-slave-p5bw krb5kdc[9760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639393, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:53.851682 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:53.843473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53460 (local address 127.6.152.62:37545)
0928 23:49:53.843633 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:53.843639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:53.843756 (+ 117us) server_negotiation.cc:407] Connection header received
0928 23:49:53.843872 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:53.843874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:53.843925 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:53.843997 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:53.844702 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:53.845290 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:53.846240 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:53.846462 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:53.846553 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:53.848987 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:53.848990 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:53.848993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:53.849017 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:53.850194 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:53.850552 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:53.850555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:53.850556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:53.850626 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:53.850839 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:53.850841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:53.850842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:53.851063 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:53.851165 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:53.851392 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 23:49:53.851526 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 23:49:53.852206 9805 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53460
I20260928 23:49:53.853122 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 9867
I20260928 23:49:53.858933 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 10001
I20260928 23:49:53.864411 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 10135
I20260928 23:49:53.869529 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 9773
2026-09-28T23:49:53Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2126 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:49:53 dist-test-slave-p5bw krb5kdc[10276](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:49:53 dist-test-slave-p5bw krb5kdc[10276](info): set up 2 sockets
Sep 28 23:49:53 dist-test-slave-p5bw krb5kdc[10276](info): commencing operation
krb5kdc: starting...
W20260928 23:49:54.901568 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:49:54 dist-test-slave-p5bw krb5kdc[10276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:49:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:49:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:49:55.022444 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:45491
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:34593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:45491
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:55.136606 10289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:55.136914 10289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:55.136992 10289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:55.140969 10289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:49:55.141017 10289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:55.141036 10289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:55.141086 10289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:49:55.141117 10289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:49:55.146301 10289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:34593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:45491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:45491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:55.147330 10289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:55.148332 10289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:55.156603 10295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:55.156610 10297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:55.156644 10289 server_base.cc:1034] running on GCE node
W20260928 23:49:55.156661 10294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:55.157361 10289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:55.158463 10289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:55.159646 10289 hybrid_clock.cc:648] HybridClock initialized: now 1790639395159637 us; error 28 us; skew 500 ppm
Sep 28 23:49:55 dist-test-slave-p5bw krb5kdc[10276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639395, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:55.162330 10289 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:49:55.163156 10289 webserver.cc:460] Webserver started at http://127.6.152.62:40225/ using document root <none> and password file <none>
I20260928 23:49:55.163715 10289 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:55.163805 10289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:55.164034 10289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:55.166040 10289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "07b3dd9836f041cfa723acc522ecca43"
format_stamp: "Formatted at 2026-09-28 23:49:55 on dist-test-slave-p5bw"
server_key: "b63dd6b29eef99022c1f923a6073d0b5"
server_key_iv: "df41cd1535102e6eb5c938e168acd829"
server_key_version: "encryptionkey@0"
I20260928 23:49:55.166611 10289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "07b3dd9836f041cfa723acc522ecca43"
format_stamp: "Formatted at 2026-09-28 23:49:55 on dist-test-slave-p5bw"
server_key: "b63dd6b29eef99022c1f923a6073d0b5"
server_key_iv: "df41cd1535102e6eb5c938e168acd829"
server_key_version: "encryptionkey@0"
I20260928 23:49:55.170207 10289 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:49:55.172499 10304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:55.174016 10289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:49:55.174158 10289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "07b3dd9836f041cfa723acc522ecca43"
format_stamp: "Formatted at 2026-09-28 23:49:55 on dist-test-slave-p5bw"
server_key: "b63dd6b29eef99022c1f923a6073d0b5"
server_key_iv: "df41cd1535102e6eb5c938e168acd829"
server_key_version: "encryptionkey@0"
I20260928 23:49:55.174279 10289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:55.186800 10289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:55.189815 10289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:55.190013 10289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:55.196995 10289 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:45491
I20260928 23:49:55.197059 10356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:45491 every 8 connection(s)
I20260928 23:49:55.197930 10289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:49:55.198712 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 10289
I20260928 23:49:55.198843 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:49:55.199110 6752 external_mini_cluster.cc:1428] Setting key 9c17fc98b4c5b3280635b8104a59fa9f
I20260928 23:49:55.201009 10357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:55.206478 10357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07b3dd9836f041cfa723acc522ecca43: Bootstrap starting.
Sep 28 23:49:55 dist-test-slave-p5bw krb5kdc[10276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:55.209080 10357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07b3dd9836f041cfa723acc522ecca43: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:55.210011 10357 log.cc:824] T 00000000000000000000000000000000 P 07b3dd9836f041cfa723acc522ecca43: Log is configured to *not* fsync() on all Append() calls
I20260928 23:49:55.212234 10357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07b3dd9836f041cfa723acc522ecca43: No bootstrap required, opened a new log
I20260928 23:49:55.213510 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:55.200478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38638 (local address 127.6.152.62:45491)
0928 23:49:55.200977 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:55.200998 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:55.201034 (+ 36us) server_negotiation.cc:407] Connection header received
0928 23:49:55.201690 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:55.201706 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:55.202038 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:55.202372 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:55.203217 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:55.204182 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:55.205097 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:55.205556 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:55.205735 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:55.208485 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:55.208492 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:55.208508 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:55.208537 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:55.210542 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:55.211000 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:55.211005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:55.211011 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:55.211143 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:55.211391 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:55.211394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:55.211396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:55.211890 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:55.212076 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:55.212286 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 23:49:55.212586 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":145,"threads_started":1}
I20260928 23:49:55.216287 10357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07b3dd9836f041cfa723acc522ecca43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07b3dd9836f041cfa723acc522ecca43" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45491 } }
I20260928 23:49:55.216498 10357 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07b3dd9836f041cfa723acc522ecca43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:55.216539 10357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07b3dd9836f041cfa723acc522ecca43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b3dd9836f041cfa723acc522ecca43, State: Initialized, Role: FOLLOWER
I20260928 23:49:55.217118 10357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07b3dd9836f041cfa723acc522ecca43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07b3dd9836f041cfa723acc522ecca43" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45491 } }
I20260928 23:49:55.217270 10357 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07b3dd9836f041cfa723acc522ecca43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:49:55.217321 10357 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07b3dd9836f041cfa723acc522ecca43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:49:55.217401 10357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07b3dd9836f041cfa723acc522ecca43 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:55.218330 10357 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07b3dd9836f041cfa723acc522ecca43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07b3dd9836f041cfa723acc522ecca43" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45491 } }
I20260928 23:49:55.218705 10357 leader_election.cc:302] T 00000000000000000000000000000000 P 07b3dd9836f041cfa723acc522ecca43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07b3dd9836f041cfa723acc522ecca43; no voters:
I20260928 23:49:55.219085 10357 leader_election.cc:288] T 00000000000000000000000000000000 P 07b3dd9836f041cfa723acc522ecca43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:49:55.219190 10362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07b3dd9836f041cfa723acc522ecca43 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:49:55.219449 10362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07b3dd9836f041cfa723acc522ecca43 [term 1 LEADER]: Becoming Leader. State: Replica: 07b3dd9836f041cfa723acc522ecca43, State: Running, Role: LEADER
I20260928 23:49:55.219728 10362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07b3dd9836f041cfa723acc522ecca43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07b3dd9836f041cfa723acc522ecca43" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45491 } }
I20260928 23:49:55.220271 10357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07b3dd9836f041cfa723acc522ecca43 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:49:55.221318 10363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07b3dd9836f041cfa723acc522ecca43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07b3dd9836f041cfa723acc522ecca43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07b3dd9836f041cfa723acc522ecca43" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45491 } } }
I20260928 23:49:55.221479 10363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07b3dd9836f041cfa723acc522ecca43 [sys.catalog]: This master's current role is: LEADER
I20260928 23:49:55.221777 10364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07b3dd9836f041cfa723acc522ecca43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07b3dd9836f041cfa723acc522ecca43. Latest consensus state: current_term: 1 leader_uuid: "07b3dd9836f041cfa723acc522ecca43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07b3dd9836f041cfa723acc522ecca43" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45491 } } }
I20260928 23:49:55.221884 10364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07b3dd9836f041cfa723acc522ecca43 [sys.catalog]: This master's current role is: LEADER
I20260928 23:49:55.221870 10371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:49:55.224913 10371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:49:55.231258 10371 catalog_manager.cc:1347] Generated new cluster ID: 78c5f10a9e184d9689ab1a244b3f113b
I20260928 23:49:55.231340 10371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:49:55.265499 10371 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:49:55.266541 10371 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:49:55.279929 10371 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07b3dd9836f041cfa723acc522ecca43: Generated new TSK 0
I20260928 23:49:55.280660 10371 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
I20260928 23:49:55.320631 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:45491
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:34593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:55.430450 10384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:55.430687 10384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:55.430729 10384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:55.434849 10384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:55.434903 10384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:55.434993 10384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:49:55.440191 10384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:34593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:45491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:55.441252 10384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:55.442243 10384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:55.451792 10389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:55.451870 10384 server_base.cc:1034] running on GCE node
W20260928 23:49:55.451810 10392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:55.451810 10390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:55.452540 10384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:55.453209 10384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:55.454363 10384 hybrid_clock.cc:648] HybridClock initialized: now 1790639395454350 us; error 27 us; skew 500 ppm
Sep 28 23:49:55 dist-test-slave-p5bw krb5kdc[10276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639395, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:55.457188 10384 init.cc:377] Logged in from keytab as kudu/127.6.152.1@KRBTEST.COM (short username kudu)
I20260928 23:49:55.457998 10384 webserver.cc:460] Webserver started at http://127.6.152.1:41329/ using document root <none> and password file <none>
I20260928 23:49:55.458505 10384 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:55.458562 10384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:55.458719 10384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:55.460647 10384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "3e31ee4f5f2c4f2c82e452cd39b09020"
format_stamp: "Formatted at 2026-09-28 23:49:55 on dist-test-slave-p5bw"
server_key: "bab5067d6cadbd336f7952a50c42e31e"
server_key_iv: "56ecf7c7d13ccdea548530ecd6f1c6ca"
server_key_version: "encryptionkey@0"
I20260928 23:49:55.461211 10384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "3e31ee4f5f2c4f2c82e452cd39b09020"
format_stamp: "Formatted at 2026-09-28 23:49:55 on dist-test-slave-p5bw"
server_key: "bab5067d6cadbd336f7952a50c42e31e"
server_key_iv: "56ecf7c7d13ccdea548530ecd6f1c6ca"
server_key_version: "encryptionkey@0"
I20260928 23:49:55.464706 10384 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:49:55.467171 10399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:55.468454 10384 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:49:55.468701 10384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "3e31ee4f5f2c4f2c82e452cd39b09020"
format_stamp: "Formatted at 2026-09-28 23:49:55 on dist-test-slave-p5bw"
server_key: "bab5067d6cadbd336f7952a50c42e31e"
server_key_iv: "56ecf7c7d13ccdea548530ecd6f1c6ca"
server_key_version: "encryptionkey@0"
I20260928 23:49:55.468849 10384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:55.485854 10384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:55.488675 10384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:55.488915 10384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:55.489496 10384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:49:55.490296 10384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:55.490372 10384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:55.490443 10384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:55.490495 10384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:55.499395 10384 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:37939
I20260928 23:49:55.499414 10513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:37939 every 8 connection(s)
I20260928 23:49:55.500241 10384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:49:55.506915 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 10384
I20260928 23:49:55.507057 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
I20260928 23:49:55.507350 6752 external_mini_cluster.cc:1428] Setting key 909f2c57468797194553788f2668c934
Sep 28 23:49:55 dist-test-slave-p5bw krb5kdc[10276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639395, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:55.515406 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:55.502079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:35649 (local address 127.6.152.62:45491)
0928 23:49:55.502236 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:55.502240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:55.503061 (+ 821us) server_negotiation.cc:407] Connection header received
0928 23:49:55.504491 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:55.504494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:55.504558 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:55.504659 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:55.505890 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:55.506437 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:55.507733 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:55.507982 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:55.508096 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:55.511464 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:55.511469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:55.511473 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:55.511509 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:55.513225 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:55.513737 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:55.513744 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:55.513749 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:55.513857 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:55.514181 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:55.514186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:55.514191 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:55.514470 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:55.514584 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:55.515057 (+ 473us) server_negotiation.cc:299] Negotiation successful
0928 23:49:55.515207 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 23:49:55.516134 10516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:55.502431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45491 (local address 127.6.152.1:35649)
0928 23:49:55.502903 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:55.502949 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:55.504265 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:55.504895 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:55.504904 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:55.505264 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:55.505732 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:55.505745 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:55.506569 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:55.506572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:55.507608 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:55.507618 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:55.508555 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:55.508558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:55.508706 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:55.509359 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:55.509375 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:55.511277 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:55.513351 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:55.513356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:55.513369 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:55.513631 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:55.513953 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:55.513955 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:55.513956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:55.514082 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:55.514576 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:55.514581 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:55.514813 (+ 232us) client_negotiation.cc:769] Sending connection context
0928 23:49:55.514966 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 23:49:55.515228 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":112,"threads_started":1}
I20260928 23:49:55.517441 10514 heartbeater.cc:347] Connected to a master server at 127.6.152.62:45491
WARNING: no policy specified for kudu/127.6.152.2@KRBTEST.COM; defaulting to no policy
I20260928 23:49:55.517700 10514 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:55.518221 10514 heartbeater.cc:511] Master 127.6.152.62:45491 requested a full tablet report, sending...
I20260928 23:49:55.519763 10320 ts_manager.cc:194] Registered new tserver with Master: 3e31ee4f5f2c4f2c82e452cd39b09020 (127.6.152.1:37939)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.2@KRBTEST.COM" created.
I20260928 23:49:55.521382 10320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.1@KRBTEST.COM'} at 127.6.152.1:35649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
I20260928 23:49:55.535455 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:45491
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:34593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:55.643038 10519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:55.643283 10519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:55.643323 10519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:55.647424 10519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:55.647481 10519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:55.647572 10519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:49:55.652992 10519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:34593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:45491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:55.654001 10519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:55.654997 10519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:55.664420 10525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:55.664474 10526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:55.664528 10519 server_base.cc:1034] running on GCE node
W20260928 23:49:55.664474 10528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:55.665161 10519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:55.665729 10519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:55.666885 10519 hybrid_clock.cc:648] HybridClock initialized: now 1790639395666875 us; error 25 us; skew 500 ppm
Sep 28 23:49:55 dist-test-slave-p5bw krb5kdc[10276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639395, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:55.669667 10519 init.cc:377] Logged in from keytab as kudu/127.6.152.2@KRBTEST.COM (short username kudu)
I20260928 23:49:55.670522 10519 webserver.cc:460] Webserver started at http://127.6.152.2:38115/ using document root <none> and password file <none>
I20260928 23:49:55.671025 10519 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:55.671079 10519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:55.671236 10519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:55.673381 10519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "4d053fb1a1c647238f57a4b6df55aef7"
format_stamp: "Formatted at 2026-09-28 23:49:55 on dist-test-slave-p5bw"
server_key: "ed8362dc2889ac07f4943ca0ef37f530"
server_key_iv: "a45757d56de8b196643b0d740eaa2c93"
server_key_version: "encryptionkey@0"
I20260928 23:49:55.673887 10519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "4d053fb1a1c647238f57a4b6df55aef7"
format_stamp: "Formatted at 2026-09-28 23:49:55 on dist-test-slave-p5bw"
server_key: "ed8362dc2889ac07f4943ca0ef37f530"
server_key_iv: "a45757d56de8b196643b0d740eaa2c93"
server_key_version: "encryptionkey@0"
I20260928 23:49:55.677654 10519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:49:55.680120 10535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:55.681435 10519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:49:55.681562 10519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "4d053fb1a1c647238f57a4b6df55aef7"
format_stamp: "Formatted at 2026-09-28 23:49:55 on dist-test-slave-p5bw"
server_key: "ed8362dc2889ac07f4943ca0ef37f530"
server_key_iv: "a45757d56de8b196643b0d740eaa2c93"
server_key_version: "encryptionkey@0"
I20260928 23:49:55.681666 10519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:55.714890 10519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:55.717953 10519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:55.718152 10519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:55.718731 10519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:49:55.719551 10519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:55.719596 10519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:55.719663 10519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:55.719700 10519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:55.728761 10519 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:39333
I20260928 23:49:55.728780 10648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:39333 every 8 connection(s)
I20260928 23:49:55.729671 10519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:49:55.731550 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 10519
I20260928 23:49:55.731685 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:49:55.732045 6752 external_mini_cluster.cc:1428] Setting key c7a948f602a3862ddebe168ac51ddf1a
Sep 28 23:49:55 dist-test-slave-p5bw krb5kdc[10276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639395, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.152.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.3@KRBTEST.COM" created.
I20260928 23:49:55.744575 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:55.731476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:50135 (local address 127.6.152.62:45491)
0928 23:49:55.731629 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:55.731634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:55.732610 (+ 976us) server_negotiation.cc:407] Connection header received
0928 23:49:55.734195 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:55.734201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:55.734263 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:55.734381 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:55.735503 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:55.736372 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:55.737723 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:55.738126 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:55.738256 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:55.741076 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:55.741083 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:55.741089 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:55.741125 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:55.742577 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:55.743076 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:55.743080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:55.743081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:55.743179 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:55.743464 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:55.743466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:55.743467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:55.743700 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:55.743774 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:55.744278 (+ 504us) server_negotiation.cc:299] Negotiation successful
0928 23:49:55.744401 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 23:49:55.745410 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:55.731949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45491 (local address 127.6.152.2:50135)
0928 23:49:55.732455 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:55.732507 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:55.733979 (+ 1472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:55.734490 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:55.734496 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:55.734838 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:55.735359 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:55.735373 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:55.736526 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:55.736529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:55.737607 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:55.737618 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:55.738353 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:55.738356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:55.738505 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:55.739130 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:55.739146 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:55.740896 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:55.742677 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:55.742682 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:55.742694 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:55.742954 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:55.743260 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:55.743262 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:55.743263 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:55.743387 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:55.743783 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:55.743788 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:55.744027 (+ 239us) client_negotiation.cc:769] Sending connection context
0928 23:49:55.744213 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 23:49:55.744554 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":158,"threads_started":1}
I20260928 23:49:55.746573 10649 heartbeater.cc:347] Connected to a master server at 127.6.152.62:45491
I20260928 23:49:55.746826 10649 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:55.747315 10649 heartbeater.cc:511] Master 127.6.152.62:45491 requested a full tablet report, sending...
I20260928 23:49:55.748399 10320 ts_manager.cc:194] Registered new tserver with Master: 4d053fb1a1c647238f57a4b6df55aef7 (127.6.152.2:39333)
I20260928 23:49:55.749241 10320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.2@KRBTEST.COM'} at 127.6.152.2:50135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
I20260928 23:49:55.761160 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:45491
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:34593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:55.870066 10654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:55.870307 10654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:55.870348 10654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:55.875098 10654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:55.875168 10654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:55.875255 10654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:49:55.883245 10654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:34593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:45491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:55.884430 10654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:55.885849 10654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:55.895706 10660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:55.895715 10662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:55.895715 10659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:55.895964 10654 server_base.cc:1034] running on GCE node
I20260928 23:49:55.896303 10654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:55.897051 10654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:55.898237 10654 hybrid_clock.cc:648] HybridClock initialized: now 1790639395898202 us; error 47 us; skew 500 ppm
Sep 28 23:49:55 dist-test-slave-p5bw krb5kdc[10276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639395, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:55.900980 10654 init.cc:377] Logged in from keytab as kudu/127.6.152.3@KRBTEST.COM (short username kudu)
I20260928 23:49:55.901743 10654 webserver.cc:460] Webserver started at http://127.6.152.3:46305/ using document root <none> and password file <none>
I20260928 23:49:55.902242 10654 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:55.902297 10654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:55.902457 10654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:55.904327 10654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "f640ac85f41349b7aaf32a053313873d"
format_stamp: "Formatted at 2026-09-28 23:49:55 on dist-test-slave-p5bw"
server_key: "f44f2e96d08a6a63d511333a36d233f7"
server_key_iv: "4540eca7065e35feb947d79d54feb46a"
server_key_version: "encryptionkey@0"
I20260928 23:49:55.904850 10654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "f640ac85f41349b7aaf32a053313873d"
format_stamp: "Formatted at 2026-09-28 23:49:55 on dist-test-slave-p5bw"
server_key: "f44f2e96d08a6a63d511333a36d233f7"
server_key_iv: "4540eca7065e35feb947d79d54feb46a"
server_key_version: "encryptionkey@0"
I20260928 23:49:55.908516 10654 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:49:55.910851 10669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:55.912077 10654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:49:55.912168 10654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "f640ac85f41349b7aaf32a053313873d"
format_stamp: "Formatted at 2026-09-28 23:49:55 on dist-test-slave-p5bw"
server_key: "f44f2e96d08a6a63d511333a36d233f7"
server_key_iv: "4540eca7065e35feb947d79d54feb46a"
server_key_version: "encryptionkey@0"
I20260928 23:49:55.912245 10654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:55.923278 10654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:55.926482 10654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:55.926626 10654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:55.927178 10654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:49:55.927920 10654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:55.927966 10654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:55.928004 10654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:55.928020 10654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:55.936710 10654 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:41573
I20260928 23:49:55.936734 10783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:41573 every 8 connection(s)
I20260928 23:49:55.937660 10654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
Sep 28 23:49:55 dist-test-slave-p5bw krb5kdc[10276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639395, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:55.947438 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 10654
I20260928 23:49:55.947570 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:49:55.947860 6752 external_mini_cluster.cc:1428] Setting key de6504bcfaa04049ff3b19101cf819dd
I20260928 23:49:55.951869 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:55.939506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:37277 (local address 127.6.152.62:45491)
0928 23:49:55.939665 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:55.939669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:55.940452 (+ 783us) server_negotiation.cc:407] Connection header received
0928 23:49:55.941859 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:55.941863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:55.941957 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:55.942061 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:55.943229 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:55.943763 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:55.944933 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:55.945165 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:55.945286 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:55.947753 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:55.947757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:55.947759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:55.947787 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:55.949663 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:55.950354 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:55.950358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:55.950360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:55.950464 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:55.950766 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:55.950768 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:55.950770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:55.950959 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:55.951042 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:55.951651 (+ 609us) server_negotiation.cc:299] Negotiation successful
0928 23:49:55.951757 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 23:49:55.952479 10786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:55.939851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45491 (local address 127.6.152.3:37277)
0928 23:49:55.940283 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:55.940324 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:55.941643 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:55.942239 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:55.942245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:55.942606 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:55.943080 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:55.943093 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:55.943888 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:55.943893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:55.944764 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:55.944773 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:55.945319 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:55.945323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:55.945503 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:55.946012 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:55.946027 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:55.947559 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:55.949793 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:55.949799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:55.949813 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:55.950217 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:55.950551 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:55.950554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:55.950555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:55.950685 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:55.951061 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:55.951066 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:55.951288 (+ 222us) client_negotiation.cc:769] Sending connection context
0928 23:49:55.951443 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 23:49:55.951682 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":115,"threads_started":1}
I20260928 23:49:55.953563 10784 heartbeater.cc:347] Connected to a master server at 127.6.152.62:45491
I20260928 23:49:55.953812 10784 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:55.954309 10784 heartbeater.cc:511] Master 127.6.152.62:45491 requested a full tablet report, sending...
I20260928 23:49:55.955261 10320 ts_manager.cc:194] Registered new tserver with Master: f640ac85f41349b7aaf32a053313873d (127.6.152.3:41573)
I20260928 23:49:55.955817 10320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.3@KRBTEST.COM'} at 127.6.152.3:37277
I20260928 23:49:55.962069 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 23:49:55 dist-test-slave-p5bw krb5kdc[10276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 23:49:55 dist-test-slave-p5bw krb5kdc[10276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.1@KRBTEST.COM
I20260928 23:49:55.988113 10791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:55.978438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50066 (local address 127.6.152.1:37939)
0928 23:49:55.978736 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:55.978749 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:55.978775 (+ 26us) server_negotiation.cc:407] Connection header received
0928 23:49:55.978825 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:55.978830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:55.979022 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:55.979143 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:55.979837 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:55.980853 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:55.981719 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:55.982104 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:55.982187 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:55.984593 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:55.984601 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:55.984613 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:55.984635 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:55.986402 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:55.986829 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:55.986835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:55.986836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:55.986919 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:55.987187 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:55.987189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:55.987190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:55.987412 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:55.987565 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:55.987716 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 23:49:55.987891 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":110,"threads_started":1}
Sep 28 23:49:55 dist-test-slave-p5bw krb5kdc[10276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:55.998838 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:55.990979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38642 (local address 127.6.152.62:45491)
0928 23:49:55.991115 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:55.991120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:55.991201 (+ 81us) server_negotiation.cc:407] Connection header received
0928 23:49:55.991342 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:55.991345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:55.991393 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:55.991483 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:55.992049 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:55.992564 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:55.993411 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:55.993647 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:55.993755 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:55.996024 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:55.996027 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:55.996029 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:55.996051 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:55.997461 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:55.997833 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:55.997837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:55.997838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:55.997913 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:55.998164 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:55.998167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:55.998168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:55.998366 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:55.998466 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:55.998633 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 23:49:55.998724 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 23:49:55.999264 10320 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38642
I20260928 23:49:55.999944 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 10384
I20260928 23:49:56.005538 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 10519
I20260928 23:49:56.010759 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 10654
I20260928 23:49:56.015687 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 10289
2026-09-28T23:49:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2146 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:49:56 dist-test-slave-p5bw krb5kdc[10796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:49:56 dist-test-slave-p5bw krb5kdc[10796](info): set up 2 sockets
Sep 28 23:49:56 dist-test-slave-p5bw krb5kdc[10796](info): commencing operation
krb5kdc: starting...
W20260928 23:49:57.077679 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:49:57 dist-test-slave-p5bw krb5kdc[10796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:49:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:49:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:49:57.204001 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:35699
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:37135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:35699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:57.309638 10809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:57.309898 10809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:57.309940 10809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:57.313807 10809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:49:57.313856 10809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:57.313875 10809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:57.313891 10809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:49:57.313906 10809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:49:57.319041 10809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:37135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:35699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:35699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:57.319954 10809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:57.320881 10809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:57.328781 10818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:57.328781 10815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:57.329037 10809 server_base.cc:1034] running on GCE node
W20260928 23:49:57.329236 10816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:57.329813 10809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:57.330832 10809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:57.331983 10809 hybrid_clock.cc:648] HybridClock initialized: now 1790639397331972 us; error 25 us; skew 500 ppm
Sep 28 23:49:57 dist-test-slave-p5bw krb5kdc[10796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639397, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:57.334589 10809 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:49:57.335276 10809 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:57.335368 10809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:57.335598 10809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:57.337607 10809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "3f8329330f8f4118b170c094160e2da6"
format_stamp: "Formatted at 2026-09-28 23:49:57 on dist-test-slave-p5bw"
server_key: "2486e7e28691890b77507d58d08ee132"
server_key_iv: "8f65f2114e963f83ff450b039f26ce45"
server_key_version: "encryptionkey@0"
I20260928 23:49:57.338152 10809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "3f8329330f8f4118b170c094160e2da6"
format_stamp: "Formatted at 2026-09-28 23:49:57 on dist-test-slave-p5bw"
server_key: "2486e7e28691890b77507d58d08ee132"
server_key_iv: "8f65f2114e963f83ff450b039f26ce45"
server_key_version: "encryptionkey@0"
I20260928 23:49:57.341674 10809 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:49:57.343910 10824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:57.345150 10809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:49:57.345278 10809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "3f8329330f8f4118b170c094160e2da6"
format_stamp: "Formatted at 2026-09-28 23:49:57 on dist-test-slave-p5bw"
server_key: "2486e7e28691890b77507d58d08ee132"
server_key_iv: "8f65f2114e963f83ff450b039f26ce45"
server_key_version: "encryptionkey@0"
I20260928 23:49:57.345387 10809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:57.360150 10809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:57.362913 10809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:57.363116 10809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:57.369860 10809 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:35699
I20260928 23:49:57.370118 10876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:35699 every 8 connection(s)
I20260928 23:49:57.370501 10809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:49:57.373171 10877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:57.378469 10877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f8329330f8f4118b170c094160e2da6: Bootstrap starting.
I20260928 23:49:57.379456 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 10809
I20260928 23:49:57.379611 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:49:57.379864 6752 external_mini_cluster.cc:1428] Setting key 0eaccdc8acbba3215d7a5772faa4cb18
I20260928 23:49:57.381126 10877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f8329330f8f4118b170c094160e2da6: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:57.382002 10877 log.cc:824] T 00000000000000000000000000000000 P 3f8329330f8f4118b170c094160e2da6: Log is configured to *not* fsync() on all Append() calls
I20260928 23:49:57.383971 10877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f8329330f8f4118b170c094160e2da6: No bootstrap required, opened a new log
I20260928 23:49:57.388056 10877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f8329330f8f4118b170c094160e2da6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8329330f8f4118b170c094160e2da6" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 35699 } }
I20260928 23:49:57.388266 10877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f8329330f8f4118b170c094160e2da6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:57.388298 10877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f8329330f8f4118b170c094160e2da6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f8329330f8f4118b170c094160e2da6, State: Initialized, Role: FOLLOWER
I20260928 23:49:57.388796 10877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f8329330f8f4118b170c094160e2da6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8329330f8f4118b170c094160e2da6" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 35699 } }
I20260928 23:49:57.388975 10877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f8329330f8f4118b170c094160e2da6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:49:57.389063 10877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f8329330f8f4118b170c094160e2da6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:49:57.389163 10877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f8329330f8f4118b170c094160e2da6 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 23:49:57 dist-test-slave-p5bw krb5kdc[10796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:57.390233 10877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f8329330f8f4118b170c094160e2da6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8329330f8f4118b170c094160e2da6" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 35699 } }
I20260928 23:49:57.390635 10877 leader_election.cc:302] T 00000000000000000000000000000000 P 3f8329330f8f4118b170c094160e2da6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f8329330f8f4118b170c094160e2da6; no voters:
I20260928 23:49:57.391019 10877 leader_election.cc:288] T 00000000000000000000000000000000 P 3f8329330f8f4118b170c094160e2da6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:49:57.391108 10882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f8329330f8f4118b170c094160e2da6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:49:57.391316 10882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f8329330f8f4118b170c094160e2da6 [term 1 LEADER]: Becoming Leader. State: Replica: 3f8329330f8f4118b170c094160e2da6, State: Running, Role: LEADER
I20260928 23:49:57.391680 10882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f8329330f8f4118b170c094160e2da6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8329330f8f4118b170c094160e2da6" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 35699 } }
I20260928 23:49:57.392305 10877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f8329330f8f4118b170c094160e2da6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:49:57.393621 10883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f8329330f8f4118b170c094160e2da6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f8329330f8f4118b170c094160e2da6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8329330f8f4118b170c094160e2da6" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 35699 } } }
I20260928 23:49:57.393738 10883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f8329330f8f4118b170c094160e2da6 [sys.catalog]: This master's current role is: LEADER
I20260928 23:49:57.394163 10884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f8329330f8f4118b170c094160e2da6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f8329330f8f4118b170c094160e2da6. Latest consensus state: current_term: 1 leader_uuid: "3f8329330f8f4118b170c094160e2da6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f8329330f8f4118b170c094160e2da6" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 35699 } } }
I20260928 23:49:57.394266 10884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f8329330f8f4118b170c094160e2da6 [sys.catalog]: This master's current role is: LEADER
I20260928 23:49:57.396194 10880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:57.380907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52320 (local address 127.6.152.62:35699)
0928 23:49:57.381381 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:57.381401 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:57.381443 (+ 42us) server_negotiation.cc:407] Connection header received
0928 23:49:57.382263 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:57.382291 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:57.382658 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:57.383021 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:57.383882 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:57.385349 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:57.386209 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:57.386919 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:57.387133 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:57.389793 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:57.389799 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:57.389815 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:57.389846 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:57.391964 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:57.392484 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:57.392489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:57.392494 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:57.392614 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:57.393560 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:57.393563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:57.393565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:57.394054 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:57.395091 (+ 1037us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:57.395250 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 23:49:57.395493 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":135,"threads_started":1}
W20260928 23:49:57.397439 10897 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f8329330f8f4118b170c094160e2da6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:49:57.397516 10897 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:49:57.397572 10891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:49:57.398430 10891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:49:57.403954 10891 catalog_manager.cc:1347] Generated new cluster ID: d37294ec32e14f208a1f20a7cfe908bb
I20260928 23:49:57.404024 10891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:49:57.419521 10891 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:49:57.420436 10891 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:49:57.430069 10891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f8329330f8f4118b170c094160e2da6: Generated new TSK 0
I20260928 23:49:57.430761 10891 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
I20260928 23:49:57.457966 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:35699
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:37135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:57.567842 10903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:57.568100 10903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:57.568146 10903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:57.572104 10903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:57.572160 10903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:57.572220 10903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:49:57.577440 10903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:37135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:35699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:57.578409 10903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:57.579417 10903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:57.588364 10908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:57.588418 10911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:57.588364 10909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:57.588584 10903 server_base.cc:1034] running on GCE node
I20260928 23:49:57.589184 10903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:57.589797 10903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:57.590982 10903 hybrid_clock.cc:648] HybridClock initialized: now 1790639397590962 us; error 42 us; skew 500 ppm
Sep 28 23:49:57 dist-test-slave-p5bw krb5kdc[10796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639397, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:57.593791 10903 init.cc:377] Logged in from keytab as kudu/127.6.152.1@KRBTEST.COM (short username kudu)
I20260928 23:49:57.594587 10903 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:57.594677 10903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:57.594901 10903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:57.596959 10903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "3769af4b674844c39d06ffdefedb9c03"
format_stamp: "Formatted at 2026-09-28 23:49:57 on dist-test-slave-p5bw"
server_key: "5f579d5c1b6536f6f36c68e0f756115f"
server_key_iv: "3501cb39f349ee480fe7c6cae943bc37"
server_key_version: "encryptionkey@0"
I20260928 23:49:57.597524 10903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "3769af4b674844c39d06ffdefedb9c03"
format_stamp: "Formatted at 2026-09-28 23:49:57 on dist-test-slave-p5bw"
server_key: "5f579d5c1b6536f6f36c68e0f756115f"
server_key_iv: "3501cb39f349ee480fe7c6cae943bc37"
server_key_version: "encryptionkey@0"
I20260928 23:49:57.601176 10903 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 23:49:57.603538 10917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:57.604936 10903 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:49:57.605077 10903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "3769af4b674844c39d06ffdefedb9c03"
format_stamp: "Formatted at 2026-09-28 23:49:57 on dist-test-slave-p5bw"
server_key: "5f579d5c1b6536f6f36c68e0f756115f"
server_key_iv: "3501cb39f349ee480fe7c6cae943bc37"
server_key_version: "encryptionkey@0"
I20260928 23:49:57.605195 10903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:57.617985 10903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:57.620904 10903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:57.621101 10903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:57.621688 10903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:49:57.622499 10903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:57.622567 10903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:57.622634 10903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:57.622684 10903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:57.631065 10903 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:39493
I20260928 23:49:57.631088 11031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:39493 every 8 connection(s)
I20260928 23:49:57.631628 10903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:49:57.633631 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 10903
I20260928 23:49:57.633747 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
I20260928 23:49:57.634069 6752 external_mini_cluster.cc:1428] Setting key 757db776314f1cdcd94642cadd7c3b75
Sep 28 23:49:57 dist-test-slave-p5bw krb5kdc[10796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639397, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.152.2@KRBTEST.COM; defaulting to no policy
I20260928 23:49:57.645742 10880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:57.633334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:38129 (local address 127.6.152.62:35699)
0928 23:49:57.633473 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:57.633477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:57.634674 (+ 1197us) server_negotiation.cc:407] Connection header received
0928 23:49:57.635908 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:57.635912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:57.635966 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:57.636077 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:57.637335 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:57.637947 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:57.639054 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:57.639294 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:57.639405 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:57.642226 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:57.642229 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:57.642231 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:57.642253 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:57.643703 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:57.644182 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:57.644186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:57.644188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:57.644267 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:57.644615 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:57.644620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:57.644621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:57.644846 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:57.644960 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:57.645451 (+ 491us) server_negotiation.cc:299] Negotiation successful
0928 23:49:57.645578 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.2@KRBTEST.COM" created.
I20260928 23:49:57.646581 11034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:57.633780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:35699 (local address 127.6.152.1:38129)
0928 23:49:57.634304 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:57.634346 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:57.635710 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:57.636241 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:57.636248 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:57.636633 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:57.637196 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:57.637210 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:57.638077 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:57.638081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:57.638925 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:57.638934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:57.639444 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:57.639448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:57.639857 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:57.640563 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:57.640580 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:57.642053 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:57.643812 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:57.643818 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:57.643834 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:57.644079 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:57.644363 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:57.644366 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:57.644368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:57.644513 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:57.644967 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:57.644976 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:57.645218 (+ 242us) client_negotiation.cc:769] Sending connection context
0928 23:49:57.645405 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 23:49:57.645737 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":151,"threads_started":1}
I20260928 23:49:57.648767 11032 heartbeater.cc:347] Connected to a master server at 127.6.152.62:35699
I20260928 23:49:57.649065 11032 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:57.649552 11032 heartbeater.cc:511] Master 127.6.152.62:35699 requested a full tablet report, sending...
I20260928 23:49:57.651257 10841 ts_manager.cc:194] Registered new tserver with Master: 3769af4b674844c39d06ffdefedb9c03 (127.6.152.1:39493)
I20260928 23:49:57.652930 10841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.1@KRBTEST.COM'} at 127.6.152.1:38129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
I20260928 23:49:57.663740 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:35699
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:37135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:57.771999 11037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:57.772262 11037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:57.772307 11037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:57.776186 11037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:57.776233 11037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:57.776292 11037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:49:57.782111 11037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:37135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:35699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:57.783104 11037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:57.784044 11037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:57.793587 11042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:57.793672 11043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:57.793587 11045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:57.793915 11037 server_base.cc:1034] running on GCE node
I20260928 23:49:57.794348 11037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:57.795002 11037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:57.796185 11037 hybrid_clock.cc:648] HybridClock initialized: now 1790639397796170 us; error 25 us; skew 500 ppm
Sep 28 23:49:57 dist-test-slave-p5bw krb5kdc[10796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639397, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:57.798833 11037 init.cc:377] Logged in from keytab as kudu/127.6.152.2@KRBTEST.COM (short username kudu)
I20260928 23:49:57.799461 11037 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:57.799520 11037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:57.799682 11037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:57.801662 11037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "6bb1ce5040b54514955b9eb608e71402"
format_stamp: "Formatted at 2026-09-28 23:49:57 on dist-test-slave-p5bw"
server_key: "9e14bfe23bc5c5d88eabaa86eb558466"
server_key_iv: "a067a2189a6294fff36bb22b1febb793"
server_key_version: "encryptionkey@0"
I20260928 23:49:57.802145 11037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "6bb1ce5040b54514955b9eb608e71402"
format_stamp: "Formatted at 2026-09-28 23:49:57 on dist-test-slave-p5bw"
server_key: "9e14bfe23bc5c5d88eabaa86eb558466"
server_key_iv: "a067a2189a6294fff36bb22b1febb793"
server_key_version: "encryptionkey@0"
I20260928 23:49:57.805568 11037 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:49:57.807709 11051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:57.808979 11037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 23:49:57.809088 11037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "6bb1ce5040b54514955b9eb608e71402"
format_stamp: "Formatted at 2026-09-28 23:49:57 on dist-test-slave-p5bw"
server_key: "9e14bfe23bc5c5d88eabaa86eb558466"
server_key_iv: "a067a2189a6294fff36bb22b1febb793"
server_key_version: "encryptionkey@0"
I20260928 23:49:57.809219 11037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:57.824288 11037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:57.827123 11037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:57.827329 11037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:57.827895 11037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:49:57.828694 11037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:57.828740 11037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:57.828805 11037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:57.828858 11037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:57.837248 11037 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:39509
I20260928 23:49:57.837256 11165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:39509 every 8 connection(s)
I20260928 23:49:57.837822 11037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:49:57.839740 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 11037
I20260928 23:49:57.839865 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:49:57.840178 6752 external_mini_cluster.cc:1428] Setting key b43e95c811efeff2a48180acc17fae4c
Sep 28 23:49:57 dist-test-slave-p5bw krb5kdc[10796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639397, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.152.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.3@KRBTEST.COM" created.
I20260928 23:49:57.852921 10880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:57.839467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:58463 (local address 127.6.152.62:35699)
0928 23:49:57.839647 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:57.839652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:57.840536 (+ 884us) server_negotiation.cc:407] Connection header received
0928 23:49:57.842233 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:57.842238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:57.842299 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:57.842392 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:57.843745 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:57.844262 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:57.846109 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:57.846322 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:57.846407 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:57.849352 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:57.849356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:57.849358 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:57.849390 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:57.850857 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:57.851382 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:57.851384 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:57.851386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:57.851459 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:57.851754 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:57.851757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:57.851758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:57.852033 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:57.852120 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:57.852592 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 23:49:57.852719 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 23:49:57.853997 11168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:57.839879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:35699 (local address 127.6.152.2:58463)
0928 23:49:57.840364 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:57.840415 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:57.842028 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:57.842513 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:57.842520 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:57.842894 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:57.843584 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:57.843606 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:57.844396 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:57.844403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:57.845980 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:57.845995 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:57.846483 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:57.846490 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:57.846741 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:57.847352 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:57.847378 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:57.849184 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:57.850963 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:57.850968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:57.850979 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:57.851281 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:57.851547 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:57.851548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:57.851550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:57.851672 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:57.852129 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:57.852134 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:57.852357 (+ 223us) client_negotiation.cc:769] Sending connection context
0928 23:49:57.852501 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 23:49:57.852917 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":140,"threads_started":1}
I20260928 23:49:57.855163 11166 heartbeater.cc:347] Connected to a master server at 127.6.152.62:35699
I20260928 23:49:57.855470 11166 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:57.855921 11166 heartbeater.cc:511] Master 127.6.152.62:35699 requested a full tablet report, sending...
I20260928 23:49:57.857048 10841 ts_manager.cc:194] Registered new tserver with Master: 6bb1ce5040b54514955b9eb608e71402 (127.6.152.2:39509)
I20260928 23:49:57.857602 10841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.2@KRBTEST.COM'} at 127.6.152.2:58463
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
I20260928 23:49:57.868342 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:35699
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:37135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:49:57.979126 11171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:57.979432 11171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:57.979522 11171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:57.983647 11171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:57.983726 11171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:57.983819 11171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:49:57.989136 11171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:37135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:35699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:57.990176 11171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:57.991176 11171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:49:58.001612 11171 server_base.cc:1034] running on GCE node
W20260928 23:49:58.001544 11176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:58.001545 11177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:58.001621 11179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:58.002142 11171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:58.002737 11171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:58.003892 11171 hybrid_clock.cc:648] HybridClock initialized: now 1790639398003879 us; error 25 us; skew 500 ppm
Sep 28 23:49:58 dist-test-slave-p5bw krb5kdc[10796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639398, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:49:58.006603 11171 init.cc:377] Logged in from keytab as kudu/127.6.152.3@KRBTEST.COM (short username kudu)
I20260928 23:49:58.007299 11171 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:58.007390 11171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:58.007616 11171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:58.009765 11171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "f8bc1a1376b84403873785373c7963f8"
format_stamp: "Formatted at 2026-09-28 23:49:58 on dist-test-slave-p5bw"
server_key: "7406c9070363923c1a646e15f5954cbf"
server_key_iv: "ddc50817816d13b0c11a3dadcd587338"
server_key_version: "encryptionkey@0"
I20260928 23:49:58.010344 11171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "f8bc1a1376b84403873785373c7963f8"
format_stamp: "Formatted at 2026-09-28 23:49:58 on dist-test-slave-p5bw"
server_key: "7406c9070363923c1a646e15f5954cbf"
server_key_iv: "ddc50817816d13b0c11a3dadcd587338"
server_key_version: "encryptionkey@0"
I20260928 23:49:58.014111 11171 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:49:58.016435 11185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:58.017678 11171 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:49:58.017809 11171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "f8bc1a1376b84403873785373c7963f8"
format_stamp: "Formatted at 2026-09-28 23:49:58 on dist-test-slave-p5bw"
server_key: "7406c9070363923c1a646e15f5954cbf"
server_key_iv: "ddc50817816d13b0c11a3dadcd587338"
server_key_version: "encryptionkey@0"
I20260928 23:49:58.017921 11171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:58.027395 11171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:58.030359 11171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:58.030560 11171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:58.031140 11171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:49:58.031956 11171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:58.032034 11171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:58.032106 11171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:58.032150 11171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:58.042610 11171 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:45147
I20260928 23:49:58.042646 11298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:45147 every 8 connection(s)
I20260928 23:49:58.043234 11171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:49:58.044250 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 11171
I20260928 23:49:58.044358 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:49:58.044629 6752 external_mini_cluster.cc:1428] Setting key 5e2ce32d2949b816304e443fdfbf6695
Sep 28 23:49:58 dist-test-slave-p5bw krb5kdc[10796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639398, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:49:58.057958 10880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:58.045586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:43509 (local address 127.6.152.62:35699)
0928 23:49:58.046913 (+ 1327us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:58.046919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:58.047015 (+ 96us) server_negotiation.cc:407] Connection header received
0928 23:49:58.047979 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:58.047982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:58.048035 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:58.048153 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:58.049461 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.050035 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.051189 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.051461 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.051579 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:58.054330 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:58.054334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:58.054337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:58.054369 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:58.055703 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:58.056213 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:58.056219 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:58.056221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:58.056306 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:58.056643 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:58.056648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:58.056649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:58.056943 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:58.057055 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:58.057653 (+ 598us) server_negotiation.cc:299] Negotiation successful
0928 23:49:58.057771 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1243}
I20260928 23:49:58.058785 11301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:58.045897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:35699 (local address 127.6.152.3:43509)
0928 23:49:58.046841 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:58.046916 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:58.047750 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:58.048360 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:58.048367 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:58.048775 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:58.049317 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:58.049330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.050163 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.050167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:58.051062 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:58.051071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.051617 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.051621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:58.051799 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:58.052516 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:49:58.052536 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:58.054151 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:58.055824 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:58.055832 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:58.055848 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:58.056105 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:58.056403 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:49:58.056407 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:49:58.056409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:49:58.056545 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:49:58.057051 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:58.057057 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:58.057371 (+ 314us) client_negotiation.cc:769] Sending connection context
0928 23:49:58.057601 (+ 230us) client_negotiation.cc:240] Negotiation successful
0928 23:49:58.057929 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":136,"threads_started":1}
I20260928 23:49:58.059886 11299 heartbeater.cc:347] Connected to a master server at 127.6.152.62:35699
I20260928 23:49:58.060191 11299 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:58.060660 11299 heartbeater.cc:511] Master 127.6.152.62:35699 requested a full tablet report, sending...
I20260928 23:49:58.061695 10841 ts_manager.cc:194] Registered new tserver with Master: f8bc1a1376b84403873785373c7963f8 (127.6.152.3:45147)
I20260928 23:49:58.062265 10841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.3@KRBTEST.COM'} at 127.6.152.3:43509
I20260928 23:49:58.070852 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:49:58.079416 10880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:58.072911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52322 (local address 127.6.152.62:35699)
0928 23:49:58.073038 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:58.073042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:58.073258 (+ 216us) server_negotiation.cc:407] Connection header received
0928 23:49:58.073366 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:58.073368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:58.073405 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:58.073476 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:58.074141 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.074628 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.075382 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.075592 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.075666 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:58.076939 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:58.076942 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:58.076944 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:49:58.076964 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:58.078142 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:58.078494 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:58.078496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:58.078497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:58.078564 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:49:58.078781 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:49:58.078783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:49:58.078784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:49:58.078969 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:58.079082 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:58.079220 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 23:49:58.079304 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:49:58.082118 10841 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:49:58.085139 10841 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:49:58.099505 11313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:58.094619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:39493 (local address 127.0.0.1:40452)
0928 23:49:58.094854 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:58.094867 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:58.094945 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:58.095675 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:58.095681 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:58.095703 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:49:58.095835 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:58.095841 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.096944 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.096949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:58.098097 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:58.098108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.098911 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.098915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:58.099042 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:58.099082 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 23:49:58.099225 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 23:49:58.099349 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":106,"threads_started":1}
I20260928 23:49:58.099538 11315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:58.094864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40452 (local address 127.6.152.1:39493)
0928 23:49:58.095195 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:58.095205 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:58.095223 (+ 18us) server_negotiation.cc:407] Connection header received
0928 23:49:58.095284 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:58.095288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:58.095524 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:58.095683 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:49:58.095965 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.096791 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.098235 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.098788 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.098912 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:58.099065 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:58.099259 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 23:49:58.099358 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":129,"threads_started":1}
I20260928 23:49:58.100211 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:58.093967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:45147 (local address 127.0.0.1:52306)
0928 23:49:58.094428 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:58.094442 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:58.094537 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:58.095880 (+ 1343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:58.095885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:58.095907 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:49:58.095991 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:58.096011 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.097456 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.097462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:58.098692 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:58.098702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.099781 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.099785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:58.099880 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:58.099909 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 23:49:58.099978 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 23:49:58.100077 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":3,"thread_start_us":94,"threads_started":1}
I20260928 23:49:58.100261 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:58.094704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52306 (local address 127.6.152.3:45147)
0928 23:49:58.095468 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:58.095478 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:58.095497 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:49:58.095576 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:58.095580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:58.095734 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:58.095903 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:49:58.096102 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.097296 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.098847 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.099677 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.099792 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:58.099948 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:58.100001 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 23:49:58.100104 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":96,"threads_started":1}
I20260928 23:49:58.100498 11312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:58.094026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54434 (local address 127.6.152.2:39509)
0928 23:49:58.095325 (+ 1299us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:58.095338 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:58.095361 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:49:58.095435 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:58.095440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:58.095645 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:58.096099 (+ 454us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:49:58.096257 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.097341 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.098913 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.099671 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.100077 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:58.100210 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:58.100256 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 23:49:58.100348 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1194,"thread_start_us":134,"threads_started":1}
I20260928 23:49:58.100857 11311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:58.093887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:39509 (local address 127.0.0.1:54434)
0928 23:49:58.094355 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:58.094400 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:58.094522 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:58.095807 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:58.095811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:58.095831 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:49:58.095938 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:58.095947 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.097496 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.097499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:58.098792 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:58.098802 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.099813 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.099816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:58.099908 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:58.099929 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 23:49:58.099982 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 23:49:58.100715 (+ 733us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":818,"thread_start_us":95,"threads_started":1}
I20260928 23:49:58.103400 11215 tablet_service.cc:1467] Processing CreateTablet for tablet b9dfe62afd9544f5af2ddd4f38db8d92 (DEFAULT_TABLE table=test-table [id=895d671a110544b9a5f9c542625064e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:58.103570 10965 tablet_service.cc:1467] Processing CreateTablet for tablet b9dfe62afd9544f5af2ddd4f38db8d92 (DEFAULT_TABLE table=test-table [id=895d671a110544b9a5f9c542625064e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:58.103623 11099 tablet_service.cc:1467] Processing CreateTablet for tablet b9dfe62afd9544f5af2ddd4f38db8d92 (DEFAULT_TABLE table=test-table [id=895d671a110544b9a5f9c542625064e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:58.104501 11099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9dfe62afd9544f5af2ddd4f38db8d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:58.104524 11215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9dfe62afd9544f5af2ddd4f38db8d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:58.104519 10965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9dfe62afd9544f5af2ddd4f38db8d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:58.110855 11316 tablet_bootstrap.cc:492] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402: Bootstrap starting.
I20260928 23:49:58.112150 11317 tablet_bootstrap.cc:492] T b9dfe62afd9544f5af2ddd4f38db8d92 P 3769af4b674844c39d06ffdefedb9c03: Bootstrap starting.
I20260928 23:49:58.112354 11318 tablet_bootstrap.cc:492] T b9dfe62afd9544f5af2ddd4f38db8d92 P f8bc1a1376b84403873785373c7963f8: Bootstrap starting.
I20260928 23:49:58.113610 11316 tablet_bootstrap.cc:654] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:58.114662 11318 tablet_bootstrap.cc:654] T b9dfe62afd9544f5af2ddd4f38db8d92 P f8bc1a1376b84403873785373c7963f8: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:58.114702 11317 tablet_bootstrap.cc:654] T b9dfe62afd9544f5af2ddd4f38db8d92 P 3769af4b674844c39d06ffdefedb9c03: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:58.114897 11316 log.cc:824] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402: Log is configured to *not* fsync() on all Append() calls
I20260928 23:49:58.115908 11317 log.cc:824] T b9dfe62afd9544f5af2ddd4f38db8d92 P 3769af4b674844c39d06ffdefedb9c03: Log is configured to *not* fsync() on all Append() calls
I20260928 23:49:58.115908 11318 log.cc:824] T b9dfe62afd9544f5af2ddd4f38db8d92 P f8bc1a1376b84403873785373c7963f8: Log is configured to *not* fsync() on all Append() calls
I20260928 23:49:58.117005 11316 tablet_bootstrap.cc:492] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402: No bootstrap required, opened a new log
I20260928 23:49:58.117192 11316 ts_tablet_manager.cc:1397] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:49:58.117830 11318 tablet_bootstrap.cc:492] T b9dfe62afd9544f5af2ddd4f38db8d92 P f8bc1a1376b84403873785373c7963f8: No bootstrap required, opened a new log
I20260928 23:49:58.117957 11317 tablet_bootstrap.cc:492] T b9dfe62afd9544f5af2ddd4f38db8d92 P 3769af4b674844c39d06ffdefedb9c03: No bootstrap required, opened a new log
I20260928 23:49:58.117983 11318 ts_tablet_manager.cc:1397] T b9dfe62afd9544f5af2ddd4f38db8d92 P f8bc1a1376b84403873785373c7963f8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:49:58.118122 11317 ts_tablet_manager.cc:1397] T b9dfe62afd9544f5af2ddd4f38db8d92 P 3769af4b674844c39d06ffdefedb9c03: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:49:58.120642 11318 raft_consensus.cc:348] T b9dfe62afd9544f5af2ddd4f38db8d92 P f8bc1a1376b84403873785373c7963f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1ce5040b54514955b9eb608e71402" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39509 } } peers { permanent_uuid: "f8bc1a1376b84403873785373c7963f8" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45147 } } peers { permanent_uuid: "3769af4b674844c39d06ffdefedb9c03" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 39493 } }
I20260928 23:49:58.121153 11318 raft_consensus.cc:374] T b9dfe62afd9544f5af2ddd4f38db8d92 P f8bc1a1376b84403873785373c7963f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:58.121268 11318 raft_consensus.cc:729] T b9dfe62afd9544f5af2ddd4f38db8d92 P f8bc1a1376b84403873785373c7963f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8bc1a1376b84403873785373c7963f8, State: Initialized, Role: FOLLOWER
I20260928 23:49:58.121413 11316 raft_consensus.cc:348] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1ce5040b54514955b9eb608e71402" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39509 } } peers { permanent_uuid: "f8bc1a1376b84403873785373c7963f8" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45147 } } peers { permanent_uuid: "3769af4b674844c39d06ffdefedb9c03" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 39493 } }
I20260928 23:49:58.121783 11316 raft_consensus.cc:374] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:58.121932 11316 raft_consensus.cc:729] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bb1ce5040b54514955b9eb608e71402, State: Initialized, Role: FOLLOWER
I20260928 23:49:58.122002 11317 raft_consensus.cc:348] T b9dfe62afd9544f5af2ddd4f38db8d92 P 3769af4b674844c39d06ffdefedb9c03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1ce5040b54514955b9eb608e71402" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39509 } } peers { permanent_uuid: "f8bc1a1376b84403873785373c7963f8" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45147 } } peers { permanent_uuid: "3769af4b674844c39d06ffdefedb9c03" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 39493 } }
I20260928 23:49:58.122246 11317 raft_consensus.cc:374] T b9dfe62afd9544f5af2ddd4f38db8d92 P 3769af4b674844c39d06ffdefedb9c03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:58.122336 11317 raft_consensus.cc:729] T b9dfe62afd9544f5af2ddd4f38db8d92 P 3769af4b674844c39d06ffdefedb9c03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3769af4b674844c39d06ffdefedb9c03, State: Initialized, Role: FOLLOWER
I20260928 23:49:58.122262 11318 consensus_queue.cc:260] T b9dfe62afd9544f5af2ddd4f38db8d92 P f8bc1a1376b84403873785373c7963f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1ce5040b54514955b9eb608e71402" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39509 } } peers { permanent_uuid: "f8bc1a1376b84403873785373c7963f8" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45147 } } peers { permanent_uuid: "3769af4b674844c39d06ffdefedb9c03" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 39493 } }
I20260928 23:49:58.122458 11316 consensus_queue.cc:260] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1ce5040b54514955b9eb608e71402" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39509 } } peers { permanent_uuid: "f8bc1a1376b84403873785373c7963f8" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45147 } } peers { permanent_uuid: "3769af4b674844c39d06ffdefedb9c03" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 39493 } }
I20260928 23:49:58.122759 11317 consensus_queue.cc:260] T b9dfe62afd9544f5af2ddd4f38db8d92 P 3769af4b674844c39d06ffdefedb9c03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1ce5040b54514955b9eb608e71402" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39509 } } peers { permanent_uuid: "f8bc1a1376b84403873785373c7963f8" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45147 } } peers { permanent_uuid: "3769af4b674844c39d06ffdefedb9c03" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 39493 } }
I20260928 23:49:58.123034 11299 heartbeater.cc:503] Master 127.6.152.62:35699 was elected leader, sending a full tablet report...
I20260928 23:49:58.123450 11318 ts_tablet_manager.cc:1428] T b9dfe62afd9544f5af2ddd4f38db8d92 P f8bc1a1376b84403873785373c7963f8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 23:49:58.123449 11316 ts_tablet_manager.cc:1428] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:49:58.123667 11317 ts_tablet_manager.cc:1428] T b9dfe62afd9544f5af2ddd4f38db8d92 P 3769af4b674844c39d06ffdefedb9c03: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260928 23:49:58.123776 11166 heartbeater.cc:503] Master 127.6.152.62:35699 was elected leader, sending a full tablet report...
I20260928 23:49:58.123952 11032 heartbeater.cc:503] Master 127.6.152.62:35699 was elected leader, sending a full tablet report...
W20260928 23:49:58.132791 11033 tablet.cc:2369] T b9dfe62afd9544f5af2ddd4f38db8d92 P 3769af4b674844c39d06ffdefedb9c03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:49:58.290019 11323 raft_consensus.cc:482] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:49:58.290264 11323 raft_consensus.cc:504] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1ce5040b54514955b9eb608e71402" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39509 } } peers { permanent_uuid: "f8bc1a1376b84403873785373c7963f8" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45147 } } peers { permanent_uuid: "3769af4b674844c39d06ffdefedb9c03" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 39493 } }
I20260928 23:49:58.291435 11323 leader_election.cc:288] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8bc1a1376b84403873785373c7963f8 (127.6.152.3:45147), 3769af4b674844c39d06ffdefedb9c03 (127.6.152.1:39493)
W20260928 23:49:58.294195 11300 tablet.cc:2369] T b9dfe62afd9544f5af2ddd4f38db8d92 P f8bc1a1376b84403873785373c7963f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:49:58.295622 11168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:58.291716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:45147 (local address 127.6.152.2:56219)
0928 23:49:58.291851 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:58.291868 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:58.291967 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:58.292227 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:58.292230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:58.292247 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:49:58.292387 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:58.292394 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.293379 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.293383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:58.294507 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:58.294518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.295194 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.295197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:58.295299 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:58.295320 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 23:49:58.295421 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 23:49:58.295492 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 23:49:58.295622 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:58.291840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:56219 (local address 127.6.152.3:45147)
0928 23:49:58.292004 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:58.292011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:58.292026 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:49:58.292086 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:58.292089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:58.292127 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:58.292215 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:49:58.292506 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.293260 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.294668 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.295068 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.295186 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:58.295248 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:58.295437 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 23:49:58.295511 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 23:49:58.295837 11325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:58.291716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:39493 (local address 127.6.152.2:54193)
0928 23:49:58.292206 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:58.292218 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:58.292394 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:58.292682 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:58.292685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:58.292701 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:49:58.292850 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:58.292856 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.293630 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.293635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:58.294724 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:58.294739 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.295404 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.295407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:58.295512 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:58.295534 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:49:58.295630 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 23:49:58.295697 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":163,"thread_start_us":166,"threads_started":1}
I20260928 23:49:58.295830 11315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:58.291838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:54193 (local address 127.6.152.1:39493)
0928 23:49:58.291968 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:58.291974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:58.292303 (+ 329us) server_negotiation.cc:407] Connection header received
0928 23:49:58.292511 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:58.292516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:58.292561 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:58.292653 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:49:58.292962 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.293523 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.294872 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.295274 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.295398 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:58.295457 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:58.295645 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 23:49:58.295715 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:49:58.296489 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9dfe62afd9544f5af2ddd4f38db8d92" candidate_uuid: "6bb1ce5040b54514955b9eb608e71402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3769af4b674844c39d06ffdefedb9c03" is_pre_election: true
I20260928 23:49:58.296531 11253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9dfe62afd9544f5af2ddd4f38db8d92" candidate_uuid: "6bb1ce5040b54514955b9eb608e71402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8bc1a1376b84403873785373c7963f8" is_pre_election: true
I20260928 23:49:58.296803 11253 raft_consensus.cc:2454] T b9dfe62afd9544f5af2ddd4f38db8d92 P f8bc1a1376b84403873785373c7963f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bb1ce5040b54514955b9eb608e71402 in term 0.
I20260928 23:49:58.296792 10986 raft_consensus.cc:2454] T b9dfe62afd9544f5af2ddd4f38db8d92 P 3769af4b674844c39d06ffdefedb9c03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bb1ce5040b54514955b9eb608e71402 in term 0.
I20260928 23:49:58.297322 11055 leader_election.cc:302] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bb1ce5040b54514955b9eb608e71402, f8bc1a1376b84403873785373c7963f8; no voters:
I20260928 23:49:58.297592 11323 raft_consensus.cc:2781] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:49:58.297673 11323 raft_consensus.cc:482] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:49:58.297703 11323 raft_consensus.cc:3037] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:58.298748 11323 raft_consensus.cc:504] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1ce5040b54514955b9eb608e71402" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39509 } } peers { permanent_uuid: "f8bc1a1376b84403873785373c7963f8" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45147 } } peers { permanent_uuid: "3769af4b674844c39d06ffdefedb9c03" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 39493 } }
I20260928 23:49:58.299116 11323 leader_election.cc:288] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402 [CANDIDATE]: Term 1 election: Requested vote from peers f8bc1a1376b84403873785373c7963f8 (127.6.152.3:45147), 3769af4b674844c39d06ffdefedb9c03 (127.6.152.1:39493)
I20260928 23:49:58.299451 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9dfe62afd9544f5af2ddd4f38db8d92" candidate_uuid: "6bb1ce5040b54514955b9eb608e71402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3769af4b674844c39d06ffdefedb9c03"
I20260928 23:49:58.299472 11253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9dfe62afd9544f5af2ddd4f38db8d92" candidate_uuid: "6bb1ce5040b54514955b9eb608e71402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8bc1a1376b84403873785373c7963f8"
I20260928 23:49:58.299546 10986 raft_consensus.cc:3037] T b9dfe62afd9544f5af2ddd4f38db8d92 P 3769af4b674844c39d06ffdefedb9c03 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:58.299642 11253 raft_consensus.cc:3037] T b9dfe62afd9544f5af2ddd4f38db8d92 P f8bc1a1376b84403873785373c7963f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:58.300575 10986 raft_consensus.cc:2454] T b9dfe62afd9544f5af2ddd4f38db8d92 P 3769af4b674844c39d06ffdefedb9c03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bb1ce5040b54514955b9eb608e71402 in term 1.
I20260928 23:49:58.300696 11253 raft_consensus.cc:2454] T b9dfe62afd9544f5af2ddd4f38db8d92 P f8bc1a1376b84403873785373c7963f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bb1ce5040b54514955b9eb608e71402 in term 1.
I20260928 23:49:58.300931 11052 leader_election.cc:302] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3769af4b674844c39d06ffdefedb9c03, 6bb1ce5040b54514955b9eb608e71402; no voters:
I20260928 23:49:58.301092 11323 raft_consensus.cc:2781] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:49:58.301294 11323 raft_consensus.cc:686] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402 [term 1 LEADER]: Becoming Leader. State: Replica: 6bb1ce5040b54514955b9eb608e71402, State: Running, Role: LEADER
I20260928 23:49:58.301604 11323 consensus_queue.cc:237] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1ce5040b54514955b9eb608e71402" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39509 } } peers { permanent_uuid: "f8bc1a1376b84403873785373c7963f8" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45147 } } peers { permanent_uuid: "3769af4b674844c39d06ffdefedb9c03" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 39493 } }
I20260928 23:49:58.304605 10841 catalog_manager.cc:5495] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bb1ce5040b54514955b9eb608e71402 (127.6.152.2). New cstate: current_term: 1 leader_uuid: "6bb1ce5040b54514955b9eb608e71402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb1ce5040b54514955b9eb608e71402" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8bc1a1376b84403873785373c7963f8" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3769af4b674844c39d06ffdefedb9c03" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 39493 } health_report { overall_health: UNKNOWN } } }
W20260928 23:49:58.338805 11167 tablet.cc:2369] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:49:58.340271 11312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:58.336652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54448 (local address 127.6.152.2:39509)
0928 23:49:58.336798 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:58.336825 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:58.336842 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:49:58.336963 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:58.336965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:58.337038 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:58.337154 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:49:58.337425 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.337919 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.338786 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:58.339068 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:58.339179 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:58.339495 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:49:58.339891 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:49:58.340032 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:58.340069 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 23:49:58.340119 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 23:49:58.345726 11253 raft_consensus.cc:1261] T b9dfe62afd9544f5af2ddd4f38db8d92 P f8bc1a1376b84403873785373c7963f8 [term 1 FOLLOWER]: Refusing update from remote peer 6bb1ce5040b54514955b9eb608e71402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:49:58.345726 10986 raft_consensus.cc:1261] T b9dfe62afd9544f5af2ddd4f38db8d92 P 3769af4b674844c39d06ffdefedb9c03 [term 1 FOLLOWER]: Refusing update from remote peer 6bb1ce5040b54514955b9eb608e71402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:49:58.346400 11323 consensus_queue.cc:1035] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3769af4b674844c39d06ffdefedb9c03" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 39493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:49:58.346598 11326 consensus_queue.cc:1035] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8bc1a1376b84403873785373c7963f8" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:49:58.357731 11329 mvcc.cc:205] Tried to move back new op lower bound from 7334458975617097728 to 7334458975445934080. Current Snapshot: MvccSnapshot[applied={T|T < 7334458975617097728 or (T in {7334458975617097728})}]
I20260928 23:49:58.363421 10841 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:49:58.363565 10841 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:49:58.365883 10841 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f8329330f8f4118b170c094160e2da6: Sending DeleteTablet for 3 replicas of tablet b9dfe62afd9544f5af2ddd4f38db8d92
I20260928 23:49:58.366607 11215 tablet_service.cc:1514] Processing DeleteTablet for tablet b9dfe62afd9544f5af2ddd4f38db8d92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:49:58 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:52306
I20260928 23:49:58.366636 11099 tablet_service.cc:1514] Processing DeleteTablet for tablet b9dfe62afd9544f5af2ddd4f38db8d92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:49:58 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:54434
I20260928 23:49:58.366729 10965 tablet_service.cc:1514] Processing DeleteTablet for tablet b9dfe62afd9544f5af2ddd4f38db8d92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:49:58 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:40452
I20260928 23:49:58.367069 11339 tablet_replica.cc:330] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402: stopping tablet replica
I20260928 23:49:58.367229 11340 tablet_replica.cc:330] T b9dfe62afd9544f5af2ddd4f38db8d92 P f8bc1a1376b84403873785373c7963f8: stopping tablet replica
I20260928 23:49:58.367409 11339 raft_consensus.cc:2229] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:49:58.367475 11341 tablet_replica.cc:330] T b9dfe62afd9544f5af2ddd4f38db8d92 P 3769af4b674844c39d06ffdefedb9c03: stopping tablet replica
I20260928 23:49:58.367764 11341 raft_consensus.cc:2229] T b9dfe62afd9544f5af2ddd4f38db8d92 P 3769af4b674844c39d06ffdefedb9c03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:49:58.367795 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 10903
I20260928 23:49:58.368100 11341 raft_consensus.cc:2258] T b9dfe62afd9544f5af2ddd4f38db8d92 P 3769af4b674844c39d06ffdefedb9c03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:49:58.368593 11340 raft_consensus.cc:2229] T b9dfe62afd9544f5af2ddd4f38db8d92 P f8bc1a1376b84403873785373c7963f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:49:58.368886 11340 raft_consensus.cc:2258] T b9dfe62afd9544f5af2ddd4f38db8d92 P f8bc1a1376b84403873785373c7963f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:49:58.368916 11339 raft_consensus.cc:2258] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:49:58.370316 11340 ts_tablet_manager.cc:1905] T b9dfe62afd9544f5af2ddd4f38db8d92 P f8bc1a1376b84403873785373c7963f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:49:58.371845 11339 ts_tablet_manager.cc:1905] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:49:58.372918 11340 ts_tablet_manager.cc:1918] T b9dfe62afd9544f5af2ddd4f38db8d92 P f8bc1a1376b84403873785373c7963f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:49:58.373018 11340 log.cc:1196] T b9dfe62afd9544f5af2ddd4f38db8d92 P f8bc1a1376b84403873785373c7963f8: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/b9dfe62afd9544f5af2ddd4f38db8d92
I20260928 23:49:58.373333 11340 ts_tablet_manager.cc:1939] T b9dfe62afd9544f5af2ddd4f38db8d92 P f8bc1a1376b84403873785373c7963f8: Deleting consensus metadata
I20260928 23:49:58.374460 10828 catalog_manager.cc:4841] TS f8bc1a1376b84403873785373c7963f8 (127.6.152.3:45147): tablet b9dfe62afd9544f5af2ddd4f38db8d92 (table test-table [id=895d671a110544b9a5f9c542625064e5]) successfully deleted
I20260928 23:49:58.374667 11339 ts_tablet_manager.cc:1918] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:49:58.374755 11339 log.cc:1196] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639374546653-6752-0/minicluster-data/ts-1/wal/wals/b9dfe62afd9544f5af2ddd4f38db8d92
I20260928 23:49:58.375051 11339 ts_tablet_manager.cc:1939] T b9dfe62afd9544f5af2ddd4f38db8d92 P 6bb1ce5040b54514955b9eb608e71402: Deleting consensus metadata
I20260928 23:49:58.376055 10825 catalog_manager.cc:4841] TS 6bb1ce5040b54514955b9eb608e71402 (127.6.152.2:39509): tablet b9dfe62afd9544f5af2ddd4f38db8d92 (table test-table [id=895d671a110544b9a5f9c542625064e5]) successfully deleted
W20260928 23:49:58.380744 10825 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.152.1:39493 (error 108)
I20260928 23:49:58.381717 11313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:58.381439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:39493 (local address 127.0.0.1:40456)
0928 23:49:58.381571 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:58.381639 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.1:39493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 23:49:58.381839 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 11037
W20260928 23:49:58.381886 10825 catalog_manager.cc:4584] TS 3769af4b674844c39d06ffdefedb9c03 (127.6.152.1:39493): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b9dfe62afd9544f5af2ddd4f38db8d92: Network error: Client connection negotiation failed: client connection to 127.6.152.1:39493: connect: Connection refused (error 111)
I20260928 23:49:58.387753 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 11171
I20260928 23:49:58.393266 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 10809
2026-09-28T23:49:58Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2377 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T23:49:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:49:58Z Disabled control of system clock
I20260928 23:49:58.423151 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:41409
--webserver_interface=127.6.152.62
--webserver_port=0
--builtin_ntp_servers=127.6.152.20:43315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:41409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 23:49:58.537138 11346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:58.537391 11346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:58.537427 11346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:58.541366 11346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:49:58.541419 11346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:58.541435 11346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:58.541453 11346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:49:58.541468 11346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:49:58.546120 11346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:41409
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:41409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:58.547070 11346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:58.548327 11346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:58.556435 11354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:58.556497 11351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:58.556435 11352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:58.556670 11346 server_base.cc:1034] running on GCE node
I20260928 23:49:58.557228 11346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:58.558300 11346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:58.559463 11346 hybrid_clock.cc:648] HybridClock initialized: now 1790639398559447 us; error 29 us; skew 500 ppm
I20260928 23:49:58.561395 11346 webserver.cc:460] Webserver started at http://127.6.152.62:40167/ using document root <none> and password file <none>
I20260928 23:49:58.561954 11346 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:58.562014 11346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:58.562242 11346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:58.564234 11346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "0a287a93a81741d290f618bfdd787c7b"
format_stamp: "Formatted at 2026-09-28 23:49:58 on dist-test-slave-p5bw"
server_key: "e5ff19b48fdadb13f757a8207baa4f4f"
server_key_iv: "141d5bea5c7a8549c31e62ec3523d77b"
server_key_version: "encryptionkey@0"
I20260928 23:49:58.564785 11346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "0a287a93a81741d290f618bfdd787c7b"
format_stamp: "Formatted at 2026-09-28 23:49:58 on dist-test-slave-p5bw"
server_key: "e5ff19b48fdadb13f757a8207baa4f4f"
server_key_iv: "141d5bea5c7a8549c31e62ec3523d77b"
server_key_version: "encryptionkey@0"
I20260928 23:49:58.568382 11346 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:49:58.570808 11360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:58.572002 11346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:49:58.572124 11346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "0a287a93a81741d290f618bfdd787c7b"
format_stamp: "Formatted at 2026-09-28 23:49:58 on dist-test-slave-p5bw"
server_key: "e5ff19b48fdadb13f757a8207baa4f4f"
server_key_iv: "141d5bea5c7a8549c31e62ec3523d77b"
server_key_version: "encryptionkey@0"
I20260928 23:49:58.572224 11346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:58.577150 11346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:58.577739 11346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:58.577929 11346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:58.584923 11412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:41409 every 8 connection(s)
I20260928 23:49:58.584925 11346 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:41409
I20260928 23:49:58.585896 11346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:49:58.588129 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 11346
I20260928 23:49:58.588249 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:49:58.588562 6752 external_mini_cluster.cc:1428] Setting key cfd5339ea5f0f139dd7d820a51806565
I20260928 23:49:58.589269 11413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:58.594084 11416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:58.589979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40182 (local address 127.6.152.62:41409)
0928 23:49:58.590623 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:58.590637 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:58.590671 (+ 34us) server_negotiation.cc:407] Connection header received
0928 23:49:58.591306 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:58.591323 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:58.591654 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:58.591979 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:58.592788 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:58.592792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:58.592842 (+ 50us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:49:58.592855 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:58.592927 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:49:58.592945 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:58.593102 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:58.593173 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 23:49:58.593419 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":135,"threads_started":1}
I20260928 23:49:58.595768 11413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a287a93a81741d290f618bfdd787c7b: Bootstrap starting.
I20260928 23:49:58.598294 11413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a287a93a81741d290f618bfdd787c7b: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:58.598997 11413 log.cc:824] T 00000000000000000000000000000000 P 0a287a93a81741d290f618bfdd787c7b: Log is configured to *not* fsync() on all Append() calls
I20260928 23:49:58.601011 11413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a287a93a81741d290f618bfdd787c7b: No bootstrap required, opened a new log
I20260928 23:49:58.604640 11413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a287a93a81741d290f618bfdd787c7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a287a93a81741d290f618bfdd787c7b" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41409 } }
I20260928 23:49:58.604909 11413 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a287a93a81741d290f618bfdd787c7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:58.604964 11413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a287a93a81741d290f618bfdd787c7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a287a93a81741d290f618bfdd787c7b, State: Initialized, Role: FOLLOWER
I20260928 23:49:58.605372 11413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a287a93a81741d290f618bfdd787c7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a287a93a81741d290f618bfdd787c7b" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41409 } }
I20260928 23:49:58.605499 11413 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a287a93a81741d290f618bfdd787c7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:49:58.605551 11413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a287a93a81741d290f618bfdd787c7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:49:58.605621 11413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a287a93a81741d290f618bfdd787c7b [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:58.606428 11413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a287a93a81741d290f618bfdd787c7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a287a93a81741d290f618bfdd787c7b" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41409 } }
I20260928 23:49:58.606822 11413 leader_election.cc:302] T 00000000000000000000000000000000 P 0a287a93a81741d290f618bfdd787c7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a287a93a81741d290f618bfdd787c7b; no voters:
I20260928 23:49:58.607172 11413 leader_election.cc:288] T 00000000000000000000000000000000 P 0a287a93a81741d290f618bfdd787c7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:49:58.607264 11418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a287a93a81741d290f618bfdd787c7b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:49:58.607499 11418 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a287a93a81741d290f618bfdd787c7b [term 1 LEADER]: Becoming Leader. State: Replica: 0a287a93a81741d290f618bfdd787c7b, State: Running, Role: LEADER
I20260928 23:49:58.607817 11418 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a287a93a81741d290f618bfdd787c7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a287a93a81741d290f618bfdd787c7b" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41409 } }
I20260928 23:49:58.608417 11413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a287a93a81741d290f618bfdd787c7b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:49:58.609525 11419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a287a93a81741d290f618bfdd787c7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a287a93a81741d290f618bfdd787c7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a287a93a81741d290f618bfdd787c7b" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41409 } } }
I20260928 23:49:58.609665 11419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a287a93a81741d290f618bfdd787c7b [sys.catalog]: This master's current role is: LEADER
I20260928 23:49:58.609530 11420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a287a93a81741d290f618bfdd787c7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a287a93a81741d290f618bfdd787c7b. Latest consensus state: current_term: 1 leader_uuid: "0a287a93a81741d290f618bfdd787c7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a287a93a81741d290f618bfdd787c7b" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41409 } } }
I20260928 23:49:58.609724 11420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a287a93a81741d290f618bfdd787c7b [sys.catalog]: This master's current role is: LEADER
I20260928 23:49:58.609985 11427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:49:58.613116 11427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:49:58.618474 11427 catalog_manager.cc:1347] Generated new cluster ID: f5f352dd82df4f29ba6dfa5fef9182f5
I20260928 23:49:58.618542 11427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:49:58.648278 11427 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:49:58.648702 11427 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:49:58.656363 11427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a287a93a81741d290f618bfdd787c7b: Generated new TSK 0
I20260928 23:49:58.657099 11427 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:49:58.661293 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:41409
--builtin_ntp_servers=127.6.152.20:43315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 23:49:58.769922 11437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:58.770171 11437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:58.770210 11437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:58.774235 11437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:58.774283 11437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:58.774386 11437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:49:58.779386 11437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:41409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:58.780416 11437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:58.781881 11437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:58.791023 11445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:58.791062 11443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:58.791065 11442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:58.791579 11437 server_base.cc:1034] running on GCE node
I20260928 23:49:58.791991 11437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:58.792585 11437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:58.793767 11437 hybrid_clock.cc:648] HybridClock initialized: now 1790639398793744 us; error 34 us; skew 500 ppm
I20260928 23:49:58.795519 11437 webserver.cc:460] Webserver started at http://127.6.152.1:33319/ using document root <none> and password file <none>
I20260928 23:49:58.796084 11437 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:58.796172 11437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:58.796379 11437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:58.798463 11437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "e04ee4c3a31c47038a7e662c9ff620b3"
format_stamp: "Formatted at 2026-09-28 23:49:58 on dist-test-slave-p5bw"
server_key: "38d09008f32047b5d10f8f599be048fb"
server_key_iv: "dc0d732484120f2e4107132ca65696ce"
server_key_version: "encryptionkey@0"
I20260928 23:49:58.798998 11437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "e04ee4c3a31c47038a7e662c9ff620b3"
format_stamp: "Formatted at 2026-09-28 23:49:58 on dist-test-slave-p5bw"
server_key: "38d09008f32047b5d10f8f599be048fb"
server_key_iv: "dc0d732484120f2e4107132ca65696ce"
server_key_version: "encryptionkey@0"
I20260928 23:49:58.802498 11437 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260928 23:49:58.804687 11451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:58.805840 11437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:49:58.805976 11437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "e04ee4c3a31c47038a7e662c9ff620b3"
format_stamp: "Formatted at 2026-09-28 23:49:58 on dist-test-slave-p5bw"
server_key: "38d09008f32047b5d10f8f599be048fb"
server_key_iv: "dc0d732484120f2e4107132ca65696ce"
server_key_version: "encryptionkey@0"
I20260928 23:49:58.806082 11437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:58.810570 11437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:58.811110 11437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:58.811281 11437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:58.811782 11437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:49:58.812614 11437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:58.812682 11437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:58.812747 11437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:58.812795 11437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:58.821087 11437 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:45613
I20260928 23:49:58.821110 11565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:45613 every 8 connection(s)
I20260928 23:49:58.821919 11437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:49:58.827569 11416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:58.823689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:43189 (local address 127.6.152.62:41409)
0928 23:49:58.823823 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:58.823827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:58.824850 (+ 1023us) server_negotiation.cc:407] Connection header received
0928 23:49:58.826081 (+ 1231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:58.826084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:58.826119 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:58.826191 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:58.827190 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:58.827193 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:58.827195 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:49:58.827206 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:58.827225 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:49:58.827233 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:58.827306 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:58.827422 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 23:49:58.827457 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:49:58.827805 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 11437
I20260928 23:49:58.827924 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
I20260928 23:49:58.828202 6752 external_mini_cluster.cc:1428] Setting key 12faba22d90a6d9ffb25a573b1ca62d1
I20260928 23:49:58.828617 11568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:58.824067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:41409 (local address 127.6.152.1:43189)
0928 23:49:58.824676 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:58.824714 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:58.825896 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:58.826344 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:58.826350 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:58.826435 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:58.826937 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:49:58.826939 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:58.826977 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:49:58.827053 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:58.827306 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:58.827311 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:58.827315 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:49:58.827441 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 23:49:58.827683 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":104,"threads_started":1}
I20260928 23:49:58.829802 11566 heartbeater.cc:347] Connected to a master server at 127.6.152.62:41409
I20260928 23:49:58.830021 11566 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:58.830310 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:41409
--builtin_ntp_servers=127.6.152.20:43315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 23:49:58.830417 11566 heartbeater.cc:511] Master 127.6.152.62:41409 requested a full tablet report, sending...
I20260928 23:49:58.831918 11377 ts_manager.cc:194] Registered new tserver with Master: e04ee4c3a31c47038a7e662c9ff620b3 (127.6.152.1:45613)
W20260928 23:49:58.943727 11569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:58.944083 11569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:58.944169 11569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:58.948289 11569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:58.948385 11569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:58.948483 11569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:49:58.953250 11569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:41409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:58.954372 11569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:58.955740 11569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:58.965092 11574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:58.965106 11577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:58.965088 11575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:58.965394 11569 server_base.cc:1034] running on GCE node
I20260928 23:49:58.965888 11569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:58.966575 11569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:58.967777 11569 hybrid_clock.cc:648] HybridClock initialized: now 1790639398967747 us; error 40 us; skew 500 ppm
I20260928 23:49:58.969760 11569 webserver.cc:460] Webserver started at http://127.6.152.2:36341/ using document root <none> and password file <none>
I20260928 23:49:58.970347 11569 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:58.970439 11569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:58.970674 11569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:58.972700 11569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "c876d92fe9bc419ab749c204287043cb"
format_stamp: "Formatted at 2026-09-28 23:49:58 on dist-test-slave-p5bw"
server_key: "8a8a3670d959b769a0efb47d55ee7022"
server_key_iv: "df7efa11d2b70c7648f6dd7ab56f34cd"
server_key_version: "encryptionkey@0"
I20260928 23:49:58.973317 11569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "c876d92fe9bc419ab749c204287043cb"
format_stamp: "Formatted at 2026-09-28 23:49:58 on dist-test-slave-p5bw"
server_key: "8a8a3670d959b769a0efb47d55ee7022"
server_key_iv: "df7efa11d2b70c7648f6dd7ab56f34cd"
server_key_version: "encryptionkey@0"
I20260928 23:49:58.977121 11569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 23:49:58.979434 11583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:58.980611 11569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:49:58.980741 11569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "c876d92fe9bc419ab749c204287043cb"
format_stamp: "Formatted at 2026-09-28 23:49:58 on dist-test-slave-p5bw"
server_key: "8a8a3670d959b769a0efb47d55ee7022"
server_key_iv: "df7efa11d2b70c7648f6dd7ab56f34cd"
server_key_version: "encryptionkey@0"
I20260928 23:49:58.980880 11569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:58.985426 11569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:58.986012 11569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:58.986189 11569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:58.986704 11569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:49:58.987622 11569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:58.987689 11569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:58.987754 11569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:58.987807 11569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:58.996176 11569 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:43565
I20260928 23:49:58.996205 11696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:43565 every 8 connection(s)
I20260928 23:49:58.997108 11569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:49:59.002640 11416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:58.998786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:47763 (local address 127.6.152.62:41409)
0928 23:49:58.998914 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:58.998918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:58.999906 (+ 988us) server_negotiation.cc:407] Connection header received
0928 23:49:59.001174 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:59.001176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:59.001212 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:59.001288 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:59.002260 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:59.002263 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:59.002266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:49:59.002277 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:59.002298 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:49:59.002305 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:59.002380 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:59.002500 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 23:49:59.002538 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:49:59.003417 11699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:58.999164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:41409 (local address 127.6.152.2:47763)
0928 23:49:58.999751 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:58.999788 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:59.000981 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:59.001436 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:59.001442 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:59.001524 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:59.002015 (+ 491us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:49:59.002016 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:59.002064 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:49:59.002126 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:59.002377 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:59.002383 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:59.002387 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:49:59.002508 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 23:49:59.002685 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":101,"threads_started":1}
I20260928 23:49:59.004483 11697 heartbeater.cc:347] Connected to a master server at 127.6.152.62:41409
I20260928 23:49:59.004700 11697 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:59.005125 11697 heartbeater.cc:511] Master 127.6.152.62:41409 requested a full tablet report, sending...
I20260928 23:49:59.006057 11377 ts_manager.cc:194] Registered new tserver with Master: c876d92fe9bc419ab749c204287043cb (127.6.152.2:43565)
I20260928 23:49:59.006503 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 11569
I20260928 23:49:59.006633 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:49:59.006906 6752 external_mini_cluster.cc:1428] Setting key a0a01c5af3739d438ac59e577fc45a08
I20260928 23:49:59.008705 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:41409
--builtin_ntp_servers=127.6.152.20:43315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 23:49:59.118695 11700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:59.118984 11700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:59.119066 11700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:59.123124 11700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:59.123203 11700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:59.123288 11700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:49:59.128057 11700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:41409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:59.129186 11700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:59.130532 11700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:59.140048 11708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:59.139987 11705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:59.140077 11706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:59.140341 11700 server_base.cc:1034] running on GCE node
I20260928 23:49:59.140785 11700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:59.141566 11700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:59.142756 11700 hybrid_clock.cc:648] HybridClock initialized: now 1790639399142706 us; error 51 us; skew 500 ppm
I20260928 23:49:59.145011 11700 webserver.cc:460] Webserver started at http://127.6.152.3:41291/ using document root <none> and password file <none>
I20260928 23:49:59.145571 11700 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:59.145632 11700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:59.145864 11700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:59.147872 11700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "1319ed05d5b240768002c00d1849f555"
format_stamp: "Formatted at 2026-09-28 23:49:59 on dist-test-slave-p5bw"
server_key: "414356ab196cb0a2d977d226cba459e1"
server_key_iv: "e77c1d644859cdcbd6d937fade90459d"
server_key_version: "encryptionkey@0"
I20260928 23:49:59.148424 11700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "1319ed05d5b240768002c00d1849f555"
format_stamp: "Formatted at 2026-09-28 23:49:59 on dist-test-slave-p5bw"
server_key: "414356ab196cb0a2d977d226cba459e1"
server_key_iv: "e77c1d644859cdcbd6d937fade90459d"
server_key_version: "encryptionkey@0"
I20260928 23:49:59.151969 11700 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 23:49:59.154351 11714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:59.155544 11700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:49:59.155668 11700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "1319ed05d5b240768002c00d1849f555"
format_stamp: "Formatted at 2026-09-28 23:49:59 on dist-test-slave-p5bw"
server_key: "414356ab196cb0a2d977d226cba459e1"
server_key_iv: "e77c1d644859cdcbd6d937fade90459d"
server_key_version: "encryptionkey@0"
I20260928 23:49:59.155778 11700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:59.160323 11700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:59.160983 11700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:59.161154 11700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:59.161677 11700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:49:59.162541 11700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:49:59.162585 11700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:59.162650 11700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:49:59.162683 11700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:59.170527 11700 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:36749
I20260928 23:49:59.170600 11827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:36749 every 8 connection(s)
I20260928 23:49:59.171408 11700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:49:59.174533 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 11700
I20260928 23:49:59.174657 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:49:59.174906 6752 external_mini_cluster.cc:1428] Setting key 6b697c8133469a88f35df80ce18e73cb
I20260928 23:49:59.177726 11416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:59.173173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:60867 (local address 127.6.152.62:41409)
0928 23:49:59.173302 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:59.173306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:59.174288 (+ 982us) server_negotiation.cc:407] Connection header received
0928 23:49:59.175885 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:59.175890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:59.175931 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:59.176013 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:59.177350 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:59.177353 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:59.177355 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:49:59.177366 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:59.177386 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:49:59.177394 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:59.177466 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:59.177590 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 23:49:59.177627 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 23:49:59.178566 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:59.173519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:41409 (local address 127.6.152.3:60867)
0928 23:49:59.174143 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:59.174181 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:59.175639 (+ 1458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:59.176204 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:59.176214 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:59.176327 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:59.177019 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:49:59.177021 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:59.177100 (+ 79us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:49:59.177202 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:59.177469 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:59.177475 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:59.177479 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:49:59.177607 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 23:49:59.177817 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":104,"threads_started":1}
I20260928 23:49:59.179621 11828 heartbeater.cc:347] Connected to a master server at 127.6.152.62:41409
I20260928 23:49:59.179842 11828 heartbeater.cc:464] Registering TS with master...
I20260928 23:49:59.180219 11828 heartbeater.cc:511] Master 127.6.152.62:41409 requested a full tablet report, sending...
I20260928 23:49:59.181186 11377 ts_manager.cc:194] Registered new tserver with Master: 1319ed05d5b240768002c00d1849f555 (127.6.152.3:36749)
I20260928 23:49:59.188761 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:49:59.191840 11416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:59.190569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40192 (local address 127.6.152.62:41409)
0928 23:49:59.190677 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:59.190680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:59.190858 (+ 178us) server_negotiation.cc:407] Connection header received
0928 23:49:59.190941 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:59.190943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:59.190971 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:59.191043 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:59.191583 (+ 540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:59.191586 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:59.191587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:49:59.191597 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:59.191613 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:49:59.191620 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:59.191684 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:59.191722 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 23:49:59.191752 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 23:49:59.194334 11377 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:49:59.197254 11377 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:49:59.208266 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:59.205180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:45613 (local address 127.0.0.1:50294)
0928 23:49:59.205739 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:59.205792 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:59.205894 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:59.206792 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:59.206798 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:59.206871 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:59.207392 (+ 521us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:49:59.207396 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:59.207417 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:49:59.207488 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:59.207891 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:59.207900 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:59.207905 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:49:59.208010 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 23:49:59.208104 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"spinlock_wait_cycles":2816,"thread_start_us":97,"threads_started":1}
I20260928 23:49:59.208372 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:59.205005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:36749 (local address 127.0.0.1:60468)
0928 23:49:59.205739 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:59.205792 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:59.205894 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:59.207349 (+ 1455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:59.207353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:59.207367 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:59.207724 (+ 357us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:49:59.207726 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:59.207735 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:49:59.207757 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:59.208114 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:59.208117 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:59.208120 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:49:59.208198 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 23:49:59.208226 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"spinlock_wait_cycles":7680,"thread_start_us":65,"threads_started":1}
I20260928 23:49:59.208536 11840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:59.205445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:43565 (local address 127.0.0.1:46054)
0928 23:49:59.206099 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:59.206113 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:59.206190 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:59.207372 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:59.207377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:59.207393 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:59.207879 (+ 486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:49:59.207881 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:59.207889 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:49:59.207907 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:59.208301 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:59.208304 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:59.208307 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:49:59.208376 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 23:49:59.208408 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":98,"threads_started":1}
I20260928 23:49:59.208892 11841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:59.205155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60468 (local address 127.6.152.3:36749)
0928 23:49:59.206191 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:59.206204 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:59.206228 (+ 24us) server_negotiation.cc:407] Connection header received
0928 23:49:59.206283 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:59.206289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:59.206480 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:59.206614 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:59.207869 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:59.207875 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:59.207886 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:49:59.207900 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:59.207980 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:49:59.208025 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:59.208136 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:59.208223 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 23:49:59.208301 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":924,"thread_start_us":132,"threads_started":1}
I20260928 23:49:59.209016 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:59.205681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50294 (local address 127.6.152.1:45613)
0928 23:49:59.206427 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:59.206437 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:59.206457 (+ 20us) server_negotiation.cc:407] Connection header received
0928 23:49:59.206501 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:59.206505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:59.206647 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:59.206769 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:59.207637 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:59.207646 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:59.207660 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:49:59.207671 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:59.207713 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:49:59.207743 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:59.207850 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:59.208667 (+ 817us) server_negotiation.cc:299] Negotiation successful
0928 23:49:59.208740 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":92,"threads_started":1}
I20260928 23:49:59.209885 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:59.205724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46054 (local address 127.6.152.2:43565)
0928 23:49:59.206209 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:59.206224 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:59.206251 (+ 27us) server_negotiation.cc:407] Connection header received
0928 23:49:59.206309 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:59.206314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:59.206513 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:59.206645 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:59.208066 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:59.208074 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:59.208089 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:49:59.208102 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:59.208136 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:49:59.208163 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:59.208259 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:59.208784 (+ 525us) server_negotiation.cc:299] Negotiation successful
0928 23:49:59.208903 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":120,"threads_started":1}
I20260928 23:49:59.211724 11762 tablet_service.cc:1467] Processing CreateTablet for tablet 4cab3ee0fe0b4a4d8cebe4c374284825 (DEFAULT_TABLE table=test-table [id=c0b9b2f7c1374ef9ae23a42945f1d335]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:59.211796 11500 tablet_service.cc:1467] Processing CreateTablet for tablet 4cab3ee0fe0b4a4d8cebe4c374284825 (DEFAULT_TABLE table=test-table [id=c0b9b2f7c1374ef9ae23a42945f1d335]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:59.212193 11631 tablet_service.cc:1467] Processing CreateTablet for tablet 4cab3ee0fe0b4a4d8cebe4c374284825 (DEFAULT_TABLE table=test-table [id=c0b9b2f7c1374ef9ae23a42945f1d335]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:49:59.212790 11500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cab3ee0fe0b4a4d8cebe4c374284825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:59.212788 11762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cab3ee0fe0b4a4d8cebe4c374284825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:59.213007 11631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cab3ee0fe0b4a4d8cebe4c374284825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:59.218637 11845 tablet_bootstrap.cc:492] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555: Bootstrap starting.
I20260928 23:49:59.220074 11846 tablet_bootstrap.cc:492] T 4cab3ee0fe0b4a4d8cebe4c374284825 P e04ee4c3a31c47038a7e662c9ff620b3: Bootstrap starting.
I20260928 23:49:59.220916 11845 tablet_bootstrap.cc:654] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:59.221185 11847 tablet_bootstrap.cc:492] T 4cab3ee0fe0b4a4d8cebe4c374284825 P c876d92fe9bc419ab749c204287043cb: Bootstrap starting.
I20260928 23:49:59.221709 11845 log.cc:824] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555: Log is configured to *not* fsync() on all Append() calls
I20260928 23:49:59.222222 11846 tablet_bootstrap.cc:654] T 4cab3ee0fe0b4a4d8cebe4c374284825 P e04ee4c3a31c47038a7e662c9ff620b3: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:59.223026 11846 log.cc:824] T 4cab3ee0fe0b4a4d8cebe4c374284825 P e04ee4c3a31c47038a7e662c9ff620b3: Log is configured to *not* fsync() on all Append() calls
I20260928 23:49:59.223251 11847 tablet_bootstrap.cc:654] T 4cab3ee0fe0b4a4d8cebe4c374284825 P c876d92fe9bc419ab749c204287043cb: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:59.223579 11845 tablet_bootstrap.cc:492] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555: No bootstrap required, opened a new log
I20260928 23:49:59.223776 11845 ts_tablet_manager.cc:1397] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 23:49:59.224020 11847 log.cc:824] T 4cab3ee0fe0b4a4d8cebe4c374284825 P c876d92fe9bc419ab749c204287043cb: Log is configured to *not* fsync() on all Append() calls
I20260928 23:49:59.224903 11846 tablet_bootstrap.cc:492] T 4cab3ee0fe0b4a4d8cebe4c374284825 P e04ee4c3a31c47038a7e662c9ff620b3: No bootstrap required, opened a new log
I20260928 23:49:59.225072 11846 ts_tablet_manager.cc:1397] T 4cab3ee0fe0b4a4d8cebe4c374284825 P e04ee4c3a31c47038a7e662c9ff620b3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260928 23:49:59.225864 11847 tablet_bootstrap.cc:492] T 4cab3ee0fe0b4a4d8cebe4c374284825 P c876d92fe9bc419ab749c204287043cb: No bootstrap required, opened a new log
I20260928 23:49:59.226042 11847 ts_tablet_manager.cc:1397] T 4cab3ee0fe0b4a4d8cebe4c374284825 P c876d92fe9bc419ab749c204287043cb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 23:49:59.227885 11845 raft_consensus.cc:348] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1319ed05d5b240768002c00d1849f555" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 36749 } } peers { permanent_uuid: "e04ee4c3a31c47038a7e662c9ff620b3" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 45613 } } peers { permanent_uuid: "c876d92fe9bc419ab749c204287043cb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43565 } }
I20260928 23:49:59.228132 11845 raft_consensus.cc:374] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:59.228183 11845 raft_consensus.cc:729] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1319ed05d5b240768002c00d1849f555, State: Initialized, Role: FOLLOWER
I20260928 23:49:59.228657 11845 consensus_queue.cc:260] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1319ed05d5b240768002c00d1849f555" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 36749 } } peers { permanent_uuid: "e04ee4c3a31c47038a7e662c9ff620b3" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 45613 } } peers { permanent_uuid: "c876d92fe9bc419ab749c204287043cb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43565 } }
I20260928 23:49:59.229357 11828 heartbeater.cc:503] Master 127.6.152.62:41409 was elected leader, sending a full tablet report...
I20260928 23:49:59.229683 11845 ts_tablet_manager.cc:1428] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260928 23:49:59.230134 11846 raft_consensus.cc:348] T 4cab3ee0fe0b4a4d8cebe4c374284825 P e04ee4c3a31c47038a7e662c9ff620b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1319ed05d5b240768002c00d1849f555" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 36749 } } peers { permanent_uuid: "e04ee4c3a31c47038a7e662c9ff620b3" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 45613 } } peers { permanent_uuid: "c876d92fe9bc419ab749c204287043cb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43565 } }
I20260928 23:49:59.230355 11846 raft_consensus.cc:374] T 4cab3ee0fe0b4a4d8cebe4c374284825 P e04ee4c3a31c47038a7e662c9ff620b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:59.230402 11846 raft_consensus.cc:729] T 4cab3ee0fe0b4a4d8cebe4c374284825 P e04ee4c3a31c47038a7e662c9ff620b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e04ee4c3a31c47038a7e662c9ff620b3, State: Initialized, Role: FOLLOWER
I20260928 23:49:59.230404 11847 raft_consensus.cc:348] T 4cab3ee0fe0b4a4d8cebe4c374284825 P c876d92fe9bc419ab749c204287043cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1319ed05d5b240768002c00d1849f555" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 36749 } } peers { permanent_uuid: "e04ee4c3a31c47038a7e662c9ff620b3" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 45613 } } peers { permanent_uuid: "c876d92fe9bc419ab749c204287043cb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43565 } }
I20260928 23:49:59.230652 11847 raft_consensus.cc:374] T 4cab3ee0fe0b4a4d8cebe4c374284825 P c876d92fe9bc419ab749c204287043cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:59.230707 11847 raft_consensus.cc:729] T 4cab3ee0fe0b4a4d8cebe4c374284825 P c876d92fe9bc419ab749c204287043cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c876d92fe9bc419ab749c204287043cb, State: Initialized, Role: FOLLOWER
I20260928 23:49:59.230862 11846 consensus_queue.cc:260] T 4cab3ee0fe0b4a4d8cebe4c374284825 P e04ee4c3a31c47038a7e662c9ff620b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1319ed05d5b240768002c00d1849f555" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 36749 } } peers { permanent_uuid: "e04ee4c3a31c47038a7e662c9ff620b3" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 45613 } } peers { permanent_uuid: "c876d92fe9bc419ab749c204287043cb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43565 } }
I20260928 23:49:59.231104 11847 consensus_queue.cc:260] T 4cab3ee0fe0b4a4d8cebe4c374284825 P c876d92fe9bc419ab749c204287043cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1319ed05d5b240768002c00d1849f555" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 36749 } } peers { permanent_uuid: "e04ee4c3a31c47038a7e662c9ff620b3" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 45613 } } peers { permanent_uuid: "c876d92fe9bc419ab749c204287043cb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43565 } }
I20260928 23:49:59.231794 11846 ts_tablet_manager.cc:1428] T 4cab3ee0fe0b4a4d8cebe4c374284825 P e04ee4c3a31c47038a7e662c9ff620b3: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 23:49:59.232004 11847 ts_tablet_manager.cc:1428] T 4cab3ee0fe0b4a4d8cebe4c374284825 P c876d92fe9bc419ab749c204287043cb: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260928 23:49:59.232152 11566 heartbeater.cc:503] Master 127.6.152.62:41409 was elected leader, sending a full tablet report...
I20260928 23:49:59.232344 11697 heartbeater.cc:503] Master 127.6.152.62:41409 was elected leader, sending a full tablet report...
W20260928 23:49:59.247992 11698 tablet.cc:2369] T 4cab3ee0fe0b4a4d8cebe4c374284825 P c876d92fe9bc419ab749c204287043cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:49:59.283006 11851 raft_consensus.cc:482] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:49:59.283191 11851 raft_consensus.cc:504] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1319ed05d5b240768002c00d1849f555" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 36749 } } peers { permanent_uuid: "e04ee4c3a31c47038a7e662c9ff620b3" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 45613 } } peers { permanent_uuid: "c876d92fe9bc419ab749c204287043cb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43565 } }
I20260928 23:49:59.284426 11851 leader_election.cc:288] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e04ee4c3a31c47038a7e662c9ff620b3 (127.6.152.1:45613), c876d92fe9bc419ab749c204287043cb (127.6.152.2:43565)
I20260928 23:49:59.286484 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:59.284828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:54727 (local address 127.6.152.2:43565)
0928 23:49:59.285026 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:59.285031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:59.285253 (+ 222us) server_negotiation.cc:407] Connection header received
0928 23:49:59.285405 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:59.285408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:59.285438 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:59.285520 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:59.286098 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:59.286100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:59.286102 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:49:59.286113 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:59.286130 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:49:59.286137 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:59.286204 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:59.286314 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 23:49:59.286353 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260928 23:49:59.286509 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:59.284764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:34609 (local address 127.6.152.1:45613)
0928 23:49:59.284919 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:59.284923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:59.284937 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:49:59.285053 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:59.285057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:59.285096 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:59.285199 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:59.285939 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:59.285943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:59.285946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:49:59.285960 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:59.285980 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:49:59.285989 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:59.286088 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:59.286352 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 23:49:59.286392 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 23:49:59.286516 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:59.284686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:43565 (local address 127.6.152.3:54727)
0928 23:49:59.285161 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:59.285171 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:59.285280 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:59.285537 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:59.285540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:59.285558 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:59.285951 (+ 393us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:49:59.285953 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:59.285965 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:49:59.285990 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:59.286208 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:59.286210 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:59.286212 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:49:59.286303 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 23:49:59.286339 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":145,"threads_started":1}
I20260928 23:49:59.286518 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:59.284611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:45613 (local address 127.6.152.3:34609)
0928 23:49:59.284856 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:59.284872 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:59.284952 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:59.285218 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:59.285221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:59.285236 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:49:59.285735 (+ 499us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:49:59.285738 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:49:59.285752 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:49:59.285782 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:49:59.286239 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:49:59.286241 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:49:59.286244 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:49:59.286341 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 23:49:59.286377 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260928 23:49:59.287230 11520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cab3ee0fe0b4a4d8cebe4c374284825" candidate_uuid: "1319ed05d5b240768002c00d1849f555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e04ee4c3a31c47038a7e662c9ff620b3" is_pre_election: true
I20260928 23:49:59.287452 11651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cab3ee0fe0b4a4d8cebe4c374284825" candidate_uuid: "1319ed05d5b240768002c00d1849f555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c876d92fe9bc419ab749c204287043cb" is_pre_election: true
I20260928 23:49:59.287534 11520 raft_consensus.cc:2454] T 4cab3ee0fe0b4a4d8cebe4c374284825 P e04ee4c3a31c47038a7e662c9ff620b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1319ed05d5b240768002c00d1849f555 in term 0.
I20260928 23:49:59.287735 11651 raft_consensus.cc:2454] T 4cab3ee0fe0b4a4d8cebe4c374284825 P c876d92fe9bc419ab749c204287043cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1319ed05d5b240768002c00d1849f555 in term 0.
I20260928 23:49:59.288065 11717 leader_election.cc:302] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1319ed05d5b240768002c00d1849f555, e04ee4c3a31c47038a7e662c9ff620b3; no voters:
I20260928 23:49:59.288367 11851 raft_consensus.cc:2781] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:49:59.288450 11851 raft_consensus.cc:482] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:49:59.288507 11851 raft_consensus.cc:3037] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:59.289664 11851 raft_consensus.cc:504] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1319ed05d5b240768002c00d1849f555" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 36749 } } peers { permanent_uuid: "e04ee4c3a31c47038a7e662c9ff620b3" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 45613 } } peers { permanent_uuid: "c876d92fe9bc419ab749c204287043cb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43565 } }
I20260928 23:49:59.289959 11851 leader_election.cc:288] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555 [CANDIDATE]: Term 1 election: Requested vote from peers e04ee4c3a31c47038a7e662c9ff620b3 (127.6.152.1:45613), c876d92fe9bc419ab749c204287043cb (127.6.152.2:43565)
I20260928 23:49:59.290215 11520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cab3ee0fe0b4a4d8cebe4c374284825" candidate_uuid: "1319ed05d5b240768002c00d1849f555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e04ee4c3a31c47038a7e662c9ff620b3"
I20260928 23:49:59.290297 11651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cab3ee0fe0b4a4d8cebe4c374284825" candidate_uuid: "1319ed05d5b240768002c00d1849f555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c876d92fe9bc419ab749c204287043cb"
I20260928 23:49:59.290331 11520 raft_consensus.cc:3037] T 4cab3ee0fe0b4a4d8cebe4c374284825 P e04ee4c3a31c47038a7e662c9ff620b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:59.290446 11651 raft_consensus.cc:3037] T 4cab3ee0fe0b4a4d8cebe4c374284825 P c876d92fe9bc419ab749c204287043cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:59.291353 11520 raft_consensus.cc:2454] T 4cab3ee0fe0b4a4d8cebe4c374284825 P e04ee4c3a31c47038a7e662c9ff620b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1319ed05d5b240768002c00d1849f555 in term 1.
I20260928 23:49:59.291443 11651 raft_consensus.cc:2454] T 4cab3ee0fe0b4a4d8cebe4c374284825 P c876d92fe9bc419ab749c204287043cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1319ed05d5b240768002c00d1849f555 in term 1.
I20260928 23:49:59.291663 11717 leader_election.cc:302] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1319ed05d5b240768002c00d1849f555, e04ee4c3a31c47038a7e662c9ff620b3; no voters:
I20260928 23:49:59.291826 11851 raft_consensus.cc:2781] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:49:59.292124 11851 raft_consensus.cc:686] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555 [term 1 LEADER]: Becoming Leader. State: Replica: 1319ed05d5b240768002c00d1849f555, State: Running, Role: LEADER
I20260928 23:49:59.292438 11851 consensus_queue.cc:237] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1319ed05d5b240768002c00d1849f555" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 36749 } } peers { permanent_uuid: "e04ee4c3a31c47038a7e662c9ff620b3" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 45613 } } peers { permanent_uuid: "c876d92fe9bc419ab749c204287043cb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43565 } }
I20260928 23:49:59.295603 11376 catalog_manager.cc:5495] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1319ed05d5b240768002c00d1849f555 (127.6.152.3). New cstate: current_term: 1 leader_uuid: "1319ed05d5b240768002c00d1849f555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1319ed05d5b240768002c00d1849f555" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 36749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e04ee4c3a31c47038a7e662c9ff620b3" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 45613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c876d92fe9bc419ab749c204287043cb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43565 } health_report { overall_health: UNKNOWN } } }
I20260928 23:49:59.305509 11841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:59.303998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60484 (local address 127.6.152.3:36749)
0928 23:49:59.304127 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:59.304130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:59.304196 (+ 66us) server_negotiation.cc:407] Connection header received
0928 23:49:59.304373 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:59.304375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:59.304426 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:59.304493 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:59.305238 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:59.305240 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:59.305242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:49:59.305251 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:59.305265 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:49:59.305271 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:59.305335 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:59.305385 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 23:49:59.305415 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:49:59.311440 11651 raft_consensus.cc:1261] T 4cab3ee0fe0b4a4d8cebe4c374284825 P c876d92fe9bc419ab749c204287043cb [term 1 FOLLOWER]: Refusing update from remote peer 1319ed05d5b240768002c00d1849f555: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:49:59.311440 11520 raft_consensus.cc:1261] T 4cab3ee0fe0b4a4d8cebe4c374284825 P e04ee4c3a31c47038a7e662c9ff620b3 [term 1 FOLLOWER]: Refusing update from remote peer 1319ed05d5b240768002c00d1849f555: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:49:59.312201 11851 consensus_queue.cc:1035] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c876d92fe9bc419ab749c204287043cb" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:49:59.312429 11855 consensus_queue.cc:1035] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e04ee4c3a31c47038a7e662c9ff620b3" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 45613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:49:59.320614 11860 mvcc.cc:205] Tried to move back new op lower bound from 7334458979572084736 to 7334458979504472064. Current Snapshot: MvccSnapshot[applied={T|T < 7334458979572084736}]
I20260928 23:49:59.320858 11859 mvcc.cc:205] Tried to move back new op lower bound from 7334458979572084736 to 7334458979504472064. Current Snapshot: MvccSnapshot[applied={T|T < 7334458979572084736}]
I20260928 23:49:59.328925 11377 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:49:59.329129 11377 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:49:59.331517 11377 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a287a93a81741d290f618bfdd787c7b: Sending DeleteTablet for 3 replicas of tablet 4cab3ee0fe0b4a4d8cebe4c374284825
I20260928 23:49:59.332377 11762 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cab3ee0fe0b4a4d8cebe4c374284825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:49:59 UTC) from {username='slave'} at 127.0.0.1:60468
I20260928 23:49:59.332587 11631 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cab3ee0fe0b4a4d8cebe4c374284825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:49:59 UTC) from {username='slave'} at 127.0.0.1:46054
I20260928 23:49:59.332959 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 11437
I20260928 23:49:59.333076 11500 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cab3ee0fe0b4a4d8cebe4c374284825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:49:59 UTC) from {username='slave'} at 127.0.0.1:50294
I20260928 23:49:59.335220 11870 tablet_replica.cc:330] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555: stopping tablet replica
I20260928 23:49:59.335570 11870 raft_consensus.cc:2229] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:49:59.336604 11870 raft_consensus.cc:2258] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:49:59.338229 11870 ts_tablet_manager.cc:1905] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:49:59.338562 11871 tablet_replica.cc:330] T 4cab3ee0fe0b4a4d8cebe4c374284825 P c876d92fe9bc419ab749c204287043cb: stopping tablet replica
I20260928 23:49:59.338918 11871 raft_consensus.cc:2229] T 4cab3ee0fe0b4a4d8cebe4c374284825 P c876d92fe9bc419ab749c204287043cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:49:59.339326 11871 raft_consensus.cc:2258] T 4cab3ee0fe0b4a4d8cebe4c374284825 P c876d92fe9bc419ab749c204287043cb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 23:49:59.340725 11363 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.152.1:45613 (error 108)
I20260928 23:49:59.340920 11871 ts_tablet_manager.cc:1905] T 4cab3ee0fe0b4a4d8cebe4c374284825 P c876d92fe9bc419ab749c204287043cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:49:59.341559 11870 ts_tablet_manager.cc:1918] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:49:59.341658 11870 log.cc:1196] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/4cab3ee0fe0b4a4d8cebe4c374284825
I20260928 23:49:59.342017 11870 ts_tablet_manager.cc:1939] T 4cab3ee0fe0b4a4d8cebe4c374284825 P 1319ed05d5b240768002c00d1849f555: Deleting consensus metadata
I20260928 23:49:59.342142 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 11569
I20260928 23:49:59.342202 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:59.341893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:45613 (local address 127.0.0.1:50298)
0928 23:49:59.342033 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:59.342111 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.1:45613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 23:49:59.343776 11361 catalog_manager.cc:4841] TS 1319ed05d5b240768002c00d1849f555 (127.6.152.3:36749): tablet 4cab3ee0fe0b4a4d8cebe4c374284825 (table test-table [id=c0b9b2f7c1374ef9ae23a42945f1d335]) successfully deleted
W20260928 23:49:59.344141 11363 catalog_manager.cc:4584] TS e04ee4c3a31c47038a7e662c9ff620b3 (127.6.152.1:45613): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4cab3ee0fe0b4a4d8cebe4c374284825: Network error: Client connection negotiation failed: client connection to 127.6.152.1:45613: connect: Connection refused (error 111)
I20260928 23:49:59.349479 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 11700
I20260928 23:49:59.349673 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:59.349406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:43565 (local address 127.0.0.1:46062)
0928 23:49:59.349546 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:59.349599 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.2:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 23:49:59.354934 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 11346
2026-09-28T23:49:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (960 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 23:49:59.376371 6752 mini_oidc.cc:152] Starting JWKS server
I20260928 23:49:59.378104 6752 webserver.cc:460] Webserver started at https://127.0.0.1:45277/ using document root <none> and password file <none>
I20260928 23:49:59.378290 6752 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 23:49:59.378567 6752 webserver.cc:460] Webserver started at http://127.0.0.1:46393/ using document root <none> and password file <none>
2026-09-28T23:49:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:49:59Z Disabled control of system clock
I20260928 23:49:59.387202 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:40713
--webserver_interface=127.6.152.62
--webserver_port=0
--builtin_ntp_servers=127.6.152.20:43591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:40713
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45277/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639374546653-6752-0/testchainca.cert with env {}
W20260928 23:49:59.494282 11878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:59.494544 11878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:59.494583 11878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:59.498533 11878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:49:59.498584 11878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:59.498600 11878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:59.498615 11878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:49:59.498631 11878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 23:49:59.498646 11878 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 23:49:59.498659 11878 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45277/jwks.json
I20260928 23:49:59.503997 11878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:40713
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:40713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45277/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639374546653-6752-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:59.505003 11878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:59.505921 11878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:49:59.514108 11886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:59.514272 11878 server_base.cc:1034] running on GCE node
W20260928 23:49:59.514212 11883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:59.514212 11884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:59.515113 11878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:59.516196 11878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:59.517392 11878 hybrid_clock.cc:648] HybridClock initialized: now 1790639399517382 us; error 34 us; skew 500 ppm
I20260928 23:49:59.519444 11878 webserver.cc:460] Webserver started at http://127.6.152.62:37835/ using document root <none> and password file <none>
I20260928 23:49:59.520035 11878 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:59.520128 11878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:59.520366 11878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:59.522505 11878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "5d88358ebd3b48979586df288aa3dcf3"
format_stamp: "Formatted at 2026-09-28 23:49:59 on dist-test-slave-p5bw"
server_key: "a2d3aeb7ccf725c43c5cad137964ad9e"
server_key_iv: "d1ebdef318b5600bb1f52aa05d0769f5"
server_key_version: "encryptionkey@0"
I20260928 23:49:59.523104 11878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "5d88358ebd3b48979586df288aa3dcf3"
format_stamp: "Formatted at 2026-09-28 23:49:59 on dist-test-slave-p5bw"
server_key: "a2d3aeb7ccf725c43c5cad137964ad9e"
server_key_iv: "d1ebdef318b5600bb1f52aa05d0769f5"
server_key_version: "encryptionkey@0"
I20260928 23:49:59.526760 11878 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 23:49:59.529243 11892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:59.530521 11878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:49:59.530661 11878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "5d88358ebd3b48979586df288aa3dcf3"
format_stamp: "Formatted at 2026-09-28 23:49:59 on dist-test-slave-p5bw"
server_key: "a2d3aeb7ccf725c43c5cad137964ad9e"
server_key_iv: "d1ebdef318b5600bb1f52aa05d0769f5"
server_key_version: "encryptionkey@0"
I20260928 23:49:59.530772 11878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:59.554689 11878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:59.555392 11878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:59.555591 11878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:59.562486 11878 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:40713
I20260928 23:49:59.562502 11944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:40713 every 8 connection(s)
I20260928 23:49:59.563493 11878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:49:59.566704 11945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:59.571962 11945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d88358ebd3b48979586df288aa3dcf3: Bootstrap starting.
I20260928 23:49:59.572887 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 11878
I20260928 23:49:59.573001 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:49:59.573292 6752 external_mini_cluster.cc:1428] Setting key 88f9849de6dd0fee16768739534e87b4
I20260928 23:49:59.574834 11945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d88358ebd3b48979586df288aa3dcf3: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:59.575625 11945 log.cc:824] T 00000000000000000000000000000000 P 5d88358ebd3b48979586df288aa3dcf3: Log is configured to *not* fsync() on all Append() calls
I20260928 23:49:59.577683 11945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d88358ebd3b48979586df288aa3dcf3: No bootstrap required, opened a new log
I20260928 23:49:59.581923 11945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d88358ebd3b48979586df288aa3dcf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d88358ebd3b48979586df288aa3dcf3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40713 } }
I20260928 23:49:59.582151 11945 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d88358ebd3b48979586df288aa3dcf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:59.582235 11945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d88358ebd3b48979586df288aa3dcf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d88358ebd3b48979586df288aa3dcf3, State: Initialized, Role: FOLLOWER
I20260928 23:49:59.582724 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:59.574379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53962 (local address 127.6.152.62:40713)
0928 23:49:59.575192 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:59.575224 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:59.575267 (+ 43us) server_negotiation.cc:407] Connection header received
0928 23:49:59.576045 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:59.576070 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:59.576497 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:59.576775 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:59.577186 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:59.578196 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:59.579421 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:59.579878 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:59.580012 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:59.580866 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:59.580872 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:59.580884 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:49:59.581262 (+ 378us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:59.581312 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:49:59.581332 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:59.581406 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:59.581561 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 23:49:59.581839 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":187,"threads_started":1}
I20260928 23:49:59.582706 11945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d88358ebd3b48979586df288aa3dcf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d88358ebd3b48979586df288aa3dcf3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40713 } }
I20260928 23:49:59.582897 11945 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d88358ebd3b48979586df288aa3dcf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:49:59.582967 11945 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d88358ebd3b48979586df288aa3dcf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:49:59.583062 11945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d88358ebd3b48979586df288aa3dcf3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:59.584012 11945 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d88358ebd3b48979586df288aa3dcf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d88358ebd3b48979586df288aa3dcf3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40713 } }
I20260928 23:49:59.584342 11945 leader_election.cc:302] T 00000000000000000000000000000000 P 5d88358ebd3b48979586df288aa3dcf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d88358ebd3b48979586df288aa3dcf3; no voters:
I20260928 23:49:59.584661 11945 leader_election.cc:288] T 00000000000000000000000000000000 P 5d88358ebd3b48979586df288aa3dcf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:49:59.585021 11950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d88358ebd3b48979586df288aa3dcf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:49:59.585211 11950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d88358ebd3b48979586df288aa3dcf3 [term 1 LEADER]: Becoming Leader. State: Replica: 5d88358ebd3b48979586df288aa3dcf3, State: Running, Role: LEADER
I20260928 23:49:59.585508 11950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d88358ebd3b48979586df288aa3dcf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d88358ebd3b48979586df288aa3dcf3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40713 } }
I20260928 23:49:59.585577 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:49:59.585902 11945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d88358ebd3b48979586df288aa3dcf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:49:59.587882 11951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d88358ebd3b48979586df288aa3dcf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d88358ebd3b48979586df288aa3dcf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d88358ebd3b48979586df288aa3dcf3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40713 } } }
I20260928 23:49:59.588018 11951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d88358ebd3b48979586df288aa3dcf3 [sys.catalog]: This master's current role is: LEADER
I20260928 23:49:59.588150 11952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d88358ebd3b48979586df288aa3dcf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d88358ebd3b48979586df288aa3dcf3. Latest consensus state: current_term: 1 leader_uuid: "5d88358ebd3b48979586df288aa3dcf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d88358ebd3b48979586df288aa3dcf3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40713 } } }
I20260928 23:49:59.588236 11952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d88358ebd3b48979586df288aa3dcf3 [sys.catalog]: This master's current role is: LEADER
W20260928 23:49:59.590785 11966 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d88358ebd3b48979586df288aa3dcf3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:49:59.590847 11966 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:49:59.590920 11965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:49:59.591588 11965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:49:59.597244 11965 catalog_manager.cc:1347] Generated new cluster ID: 1ff4244f575a40548d2a632b08733153
I20260928 23:49:59.597327 11965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:49:59.604472 11965 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:49:59.605729 11965 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:49:59.623423 11965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d88358ebd3b48979586df288aa3dcf3: Generated new TSK 0
I20260928 23:49:59.624323 11965 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 23:49:59.656059 11975 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 23:49:59.656184 11975 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 23:49:59.656785 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:59.636320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53970 (local address 127.6.152.62:40713)
0928 23:49:59.636464 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:59.636470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:59.636706 (+ 236us) server_negotiation.cc:407] Connection header received
0928 23:49:59.636935 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:59.636937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:59.636982 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:59.637082 (+ 100us) server_negotiation.cc:226] Negotiated authn=JWT
0928 23:49:59.637367 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:59.637895 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:59.639114 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:59.639323 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:59.639396 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:59.639758 (+ 362us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 23:49:59.656356 (+ 16598us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45277/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 23:49:59.656634 (+ 278us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53970: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45277/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 23:49:59.656888 11948 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53970: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45277/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 23:49:59.657938 11974 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:49:59.636255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40713 (local address 127.0.0.1:53970)
0928 23:49:59.636615 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:59.636632 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:59.636763 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:59.637125 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:59.637129 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:59.637138 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 23:49:59.637261 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:59.637267 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:59.637998 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:59.638002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:59.639003 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:59.639013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:59.639429 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:59.639433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:59.639534 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:59.639586 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 23:49:59.656590 (+ 17004us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45277/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 23:49:59.656783 (+ 193us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.152.62:40713: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45277/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":107,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.6.152.62:40713: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45277/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 23:49:59.658805 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 11878
2026-09-28T23:49:59Z chronyd exiting
I20260928 23:49:59.696272 6752 test_util.cc:175] -----------------------------------------------
I20260928 23:49:59.696359 6752 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639374546653-6752-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (321 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 23:49:59.697408 6752 mini_oidc.cc:152] Starting JWKS server
I20260928 23:49:59.698233 6752 webserver.cc:460] Webserver started at https://127.0.0.1:38845/ using document root <none> and password file <none>
I20260928 23:49:59.698314 6752 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 23:49:59.698536 6752 webserver.cc:460] Webserver started at http://127.0.0.1:43823/ using document root <none> and password file <none>
2026-09-28T23:49:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:49:59Z Disabled control of system clock
I20260928 23:49:59.707520 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:44673
--webserver_interface=127.6.152.62
--webserver_port=0
--builtin_ntp_servers=127.6.152.20:32769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:44673
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38845/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639374546653-6752-0/testchainca.cert with env {}
W20260928 23:49:59.813601 11981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:49:59.813846 11981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:49:59.813913 11981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:49:59.817842 11981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:49:59.817891 11981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:49:59.817906 11981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:49:59.817924 11981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:49:59.817970 11981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 23:49:59.818011 11981 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 23:49:59.818035 11981 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38845/jwks.json
I20260928 23:49:59.823294 11981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:32769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:44673
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:44673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38845/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639374546653-6752-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:49:59.824494 11981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:49:59.825536 11981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:49:59.833557 11981 server_base.cc:1034] running on GCE node
W20260928 23:49:59.833541 11987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:59.833611 11989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:49:59.833541 11986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:49:59.834345 11981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:49:59.835320 11981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:49:59.836478 11981 hybrid_clock.cc:648] HybridClock initialized: now 1790639399836464 us; error 24 us; skew 500 ppm
I20260928 23:49:59.838186 11981 webserver.cc:460] Webserver started at http://127.6.152.62:43139/ using document root <none> and password file <none>
I20260928 23:49:59.838719 11981 fs_manager.cc:359] Metadata directory not provided
I20260928 23:49:59.838788 11981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:49:59.839018 11981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:49:59.840953 11981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "e68237194c954866a752f94d43248d65"
format_stamp: "Formatted at 2026-09-28 23:49:59 on dist-test-slave-p5bw"
server_key: "97259503346b212731b0e7a913c7cec2"
server_key_iv: "a19b4a38c8dceb73be66b8a2be36dc01"
server_key_version: "encryptionkey@0"
I20260928 23:49:59.841481 11981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "e68237194c954866a752f94d43248d65"
format_stamp: "Formatted at 2026-09-28 23:49:59 on dist-test-slave-p5bw"
server_key: "97259503346b212731b0e7a913c7cec2"
server_key_iv: "a19b4a38c8dceb73be66b8a2be36dc01"
server_key_version: "encryptionkey@0"
I20260928 23:49:59.844866 11981 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:49:59.847043 11995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:49:59.848155 11981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:49:59.848272 11981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "e68237194c954866a752f94d43248d65"
format_stamp: "Formatted at 2026-09-28 23:49:59 on dist-test-slave-p5bw"
server_key: "97259503346b212731b0e7a913c7cec2"
server_key_iv: "a19b4a38c8dceb73be66b8a2be36dc01"
server_key_version: "encryptionkey@0"
I20260928 23:49:59.848372 11981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:49:59.884941 11981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:49:59.885774 11981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:49:59.885970 11981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:49:59.892590 11981 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:44673
I20260928 23:49:59.892593 12047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:44673 every 8 connection(s)
I20260928 23:49:59.893568 11981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:49:59.896576 12048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:49:59.902022 12048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68237194c954866a752f94d43248d65: Bootstrap starting.
I20260928 23:49:59.903270 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 11981
I20260928 23:49:59.903381 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:49:59.903654 6752 external_mini_cluster.cc:1428] Setting key bd0fbf291e410b0d1b9acd8339ede4e8
I20260928 23:49:59.904876 12048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e68237194c954866a752f94d43248d65: Neither blocks nor log segments found. Creating new log.
I20260928 23:49:59.905610 12048 log.cc:824] T 00000000000000000000000000000000 P e68237194c954866a752f94d43248d65: Log is configured to *not* fsync() on all Append() calls
I20260928 23:49:59.907635 12048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68237194c954866a752f94d43248d65: No bootstrap required, opened a new log
I20260928 23:49:59.911862 12048 raft_consensus.cc:348] T 00000000000000000000000000000000 P e68237194c954866a752f94d43248d65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68237194c954866a752f94d43248d65" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 44673 } }
I20260928 23:49:59.912106 12048 raft_consensus.cc:374] T 00000000000000000000000000000000 P e68237194c954866a752f94d43248d65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:49:59.912182 12048 raft_consensus.cc:729] T 00000000000000000000000000000000 P e68237194c954866a752f94d43248d65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e68237194c954866a752f94d43248d65, State: Initialized, Role: FOLLOWER
I20260928 23:49:59.912389 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:59.904718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56596 (local address 127.6.152.62:44673)
0928 23:49:59.905383 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:59.905398 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:59.905432 (+ 34us) server_negotiation.cc:407] Connection header received
0928 23:49:59.906092 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:59.906110 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:59.906426 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:59.906698 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:49:59.907018 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:59.908006 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:59.909205 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:59.909654 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:59.909764 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:59.910612 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:49:59.910616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:49:59.910629 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:49:59.910994 (+ 365us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:49:59.911044 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:49:59.911063 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:49:59.911136 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 23:49:59.911296 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 23:49:59.911592 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":129,"threads_started":1}
I20260928 23:49:59.912688 12048 consensus_queue.cc:260] T 00000000000000000000000000000000 P e68237194c954866a752f94d43248d65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68237194c954866a752f94d43248d65" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 44673 } }
I20260928 23:49:59.912853 12048 raft_consensus.cc:388] T 00000000000000000000000000000000 P e68237194c954866a752f94d43248d65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:49:59.912938 12048 raft_consensus.cc:482] T 00000000000000000000000000000000 P e68237194c954866a752f94d43248d65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:49:59.913101 12048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e68237194c954866a752f94d43248d65 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:49:59.914188 12048 raft_consensus.cc:504] T 00000000000000000000000000000000 P e68237194c954866a752f94d43248d65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68237194c954866a752f94d43248d65" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 44673 } }
I20260928 23:49:59.914561 12048 leader_election.cc:302] T 00000000000000000000000000000000 P e68237194c954866a752f94d43248d65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e68237194c954866a752f94d43248d65; no voters:
I20260928 23:49:59.914885 12048 leader_election.cc:288] T 00000000000000000000000000000000 P e68237194c954866a752f94d43248d65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:49:59.914968 12053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e68237194c954866a752f94d43248d65 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:49:59.915179 12053 raft_consensus.cc:686] T 00000000000000000000000000000000 P e68237194c954866a752f94d43248d65 [term 1 LEADER]: Becoming Leader. State: Replica: e68237194c954866a752f94d43248d65, State: Running, Role: LEADER
I20260928 23:49:59.915436 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:49:59.915464 12053 consensus_queue.cc:237] T 00000000000000000000000000000000 P e68237194c954866a752f94d43248d65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68237194c954866a752f94d43248d65" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 44673 } }
I20260928 23:49:59.916139 12048 sys_catalog.cc:564] T 00000000000000000000000000000000 P e68237194c954866a752f94d43248d65 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:49:59.917009 12054 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68237194c954866a752f94d43248d65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e68237194c954866a752f94d43248d65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68237194c954866a752f94d43248d65" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 44673 } } }
I20260928 23:49:59.917146 12054 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68237194c954866a752f94d43248d65 [sys.catalog]: This master's current role is: LEADER
I20260928 23:49:59.917478 12061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:49:59.917438 12055 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68237194c954866a752f94d43248d65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e68237194c954866a752f94d43248d65. Latest consensus state: current_term: 1 leader_uuid: "e68237194c954866a752f94d43248d65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68237194c954866a752f94d43248d65" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 44673 } } }
I20260928 23:49:59.917593 12055 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68237194c954866a752f94d43248d65 [sys.catalog]: This master's current role is: LEADER
I20260928 23:49:59.920404 12061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:49:59.926368 12061 catalog_manager.cc:1347] Generated new cluster ID: 8c0bb38a86564b8590b284959b3eaa25
I20260928 23:49:59.926442 12061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:49:59.941454 12061 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:49:59.942353 12061 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:49:59.956233 12061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e68237194c954866a752f94d43248d65: Generated new TSK 0
I20260928 23:49:59.957033 12061 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 23:49:59.988677 12078 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 23:49:59.988775 12078 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 23:49:59.989444 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:49:59.966248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56608 (local address 127.6.152.62:44673)
0928 23:49:59.966383 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 23:49:59.966388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:49:59.966589 (+ 201us) server_negotiation.cc:407] Connection header received
0928 23:49:59.966752 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:49:59.966754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:49:59.966798 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:49:59.966905 (+ 107us) server_negotiation.cc:226] Negotiated authn=JWT
0928 23:49:59.967146 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:59.967662 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:59.968908 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:49:59.969139 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:49:59.969225 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:59.969535 (+ 310us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 23:49:59.989079 (+ 19544us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 23:49:59.989331 (+ 252us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56608: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 23:49:59.989539 12051 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56608: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 23:49:59.989671 12077 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:49:59.966187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:44673 (local address 127.0.0.1:56608)
0928 23:49:59.966490 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 23:49:59.966508 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:49:59.966622 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:49:59.966920 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:49:59.966924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:49:59.966932 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 23:49:59.967047 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:59.967054 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:59.967777 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:59.967781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:59.968738 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:49:59.968749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:49:59.969253 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:49:59.969257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:49:59.969352 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:49:59.969380 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 23:49:59.989296 (+ 19916us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 23:49:59.989496 (+ 200us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.152.62:44673: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":106,"threads_started":1}
I20260928 23:49:59.990445 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 11981
2026-09-28T23:50:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (332 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 23:50:00.029917 6752 mini_oidc.cc:152] Starting JWKS server
I20260928 23:50:00.030786 6752 webserver.cc:460] Webserver started at https://127.0.0.1:39183/ using document root <none> and password file <none>
I20260928 23:50:00.030903 6752 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 23:50:00.031215 6752 webserver.cc:460] Webserver started at http://127.0.0.1:44417/ using document root <none> and password file <none>
2026-09-28T23:50:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:50:00Z Disabled control of system clock
I20260928 23:50:00.039726 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:35133
--webserver_interface=127.6.152.62
--webserver_port=0
--builtin_ntp_servers=127.6.152.20:42697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:35133
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39183/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 23:50:00.147981 12084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:00.148269 12084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:00.148341 12084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:00.152352 12084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:50:00.152434 12084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:00.152464 12084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:00.152500 12084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:50:00.152530 12084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 23:50:00.152567 12084 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 23:50:00.152596 12084 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39183/jwks.json
I20260928 23:50:00.157481 12084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:42697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:35133
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:35133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39183/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:00.158541 12084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:00.159865 12084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:00.168061 12090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:00.168128 12084 server_base.cc:1034] running on GCE node
W20260928 23:50:00.168138 12092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:00.168061 12089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:00.168895 12084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:00.169947 12084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:00.171113 12084 hybrid_clock.cc:648] HybridClock initialized: now 1790639400171093 us; error 32 us; skew 500 ppm
I20260928 23:50:00.172895 12084 webserver.cc:460] Webserver started at http://127.6.152.62:38357/ using document root <none> and password file <none>
I20260928 23:50:00.173470 12084 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:00.173533 12084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:00.173738 12084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:00.175709 12084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "fec2dfb353f443c8b10a94d83571bba5"
format_stamp: "Formatted at 2026-09-28 23:50:00 on dist-test-slave-p5bw"
server_key: "ac0dada083d944e210499bb36f8698df"
server_key_iv: "8e933eefa6e41346de554f0c8ed9ed91"
server_key_version: "encryptionkey@0"
I20260928 23:50:00.176263 12084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "fec2dfb353f443c8b10a94d83571bba5"
format_stamp: "Formatted at 2026-09-28 23:50:00 on dist-test-slave-p5bw"
server_key: "ac0dada083d944e210499bb36f8698df"
server_key_iv: "8e933eefa6e41346de554f0c8ed9ed91"
server_key_version: "encryptionkey@0"
I20260928 23:50:00.179788 12084 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 23:50:00.182159 12098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:00.183316 12084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:50:00.183451 12084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "fec2dfb353f443c8b10a94d83571bba5"
format_stamp: "Formatted at 2026-09-28 23:50:00 on dist-test-slave-p5bw"
server_key: "ac0dada083d944e210499bb36f8698df"
server_key_iv: "8e933eefa6e41346de554f0c8ed9ed91"
server_key_version: "encryptionkey@0"
I20260928 23:50:00.183562 12084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:00.200115 12084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:00.200770 12084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:00.200989 12084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:00.207464 12084 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:35133
I20260928 23:50:00.207512 12150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:35133 every 8 connection(s)
I20260928 23:50:00.208398 12084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:50:00.211591 12151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:00.215166 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 12084
I20260928 23:50:00.215261 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:50:00.215533 6752 external_mini_cluster.cc:1428] Setting key 8627878aa9f36ec83a63b19945acb2f5
I20260928 23:50:00.217168 12151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fec2dfb353f443c8b10a94d83571bba5: Bootstrap starting.
I20260928 23:50:00.219300 12151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fec2dfb353f443c8b10a94d83571bba5: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:00.220055 12151 log.cc:824] T 00000000000000000000000000000000 P fec2dfb353f443c8b10a94d83571bba5: Log is configured to *not* fsync() on all Append() calls
I20260928 23:50:00.221838 12151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fec2dfb353f443c8b10a94d83571bba5: No bootstrap required, opened a new log
I20260928 23:50:00.224367 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:00.216895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57202 (local address 127.6.152.62:35133)
0928 23:50:00.217539 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:00.217554 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:00.217588 (+ 34us) server_negotiation.cc:407] Connection header received
0928 23:50:00.218200 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:00.218216 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:00.218612 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:00.218953 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:00.219407 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:00.220421 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:00.221356 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:00.221813 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:00.221976 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:00.222811 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:00.222816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:00.222837 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:50:00.223198 (+ 361us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:00.223249 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:50:00.223267 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:00.223375 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:00.223486 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 23:50:00.223731 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":125,"threads_started":1}
I20260928 23:50:00.225328 12151 raft_consensus.cc:348] T 00000000000000000000000000000000 P fec2dfb353f443c8b10a94d83571bba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec2dfb353f443c8b10a94d83571bba5" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 35133 } }
I20260928 23:50:00.225562 12151 raft_consensus.cc:374] T 00000000000000000000000000000000 P fec2dfb353f443c8b10a94d83571bba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:00.225633 12151 raft_consensus.cc:729] T 00000000000000000000000000000000 P fec2dfb353f443c8b10a94d83571bba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec2dfb353f443c8b10a94d83571bba5, State: Initialized, Role: FOLLOWER
I20260928 23:50:00.226034 12151 consensus_queue.cc:260] T 00000000000000000000000000000000 P fec2dfb353f443c8b10a94d83571bba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec2dfb353f443c8b10a94d83571bba5" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 35133 } }
I20260928 23:50:00.226189 12151 raft_consensus.cc:388] T 00000000000000000000000000000000 P fec2dfb353f443c8b10a94d83571bba5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:50:00.226258 12151 raft_consensus.cc:482] T 00000000000000000000000000000000 P fec2dfb353f443c8b10a94d83571bba5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:50:00.226356 12151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fec2dfb353f443c8b10a94d83571bba5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:00.227113 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:50:00.227280 12151 raft_consensus.cc:504] T 00000000000000000000000000000000 P fec2dfb353f443c8b10a94d83571bba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec2dfb353f443c8b10a94d83571bba5" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 35133 } }
I20260928 23:50:00.227625 12151 leader_election.cc:302] T 00000000000000000000000000000000 P fec2dfb353f443c8b10a94d83571bba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fec2dfb353f443c8b10a94d83571bba5; no voters:
I20260928 23:50:00.227932 12151 leader_election.cc:288] T 00000000000000000000000000000000 P fec2dfb353f443c8b10a94d83571bba5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:50:00.228024 12156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fec2dfb353f443c8b10a94d83571bba5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:50:00.228216 12156 raft_consensus.cc:686] T 00000000000000000000000000000000 P fec2dfb353f443c8b10a94d83571bba5 [term 1 LEADER]: Becoming Leader. State: Replica: fec2dfb353f443c8b10a94d83571bba5, State: Running, Role: LEADER
I20260928 23:50:00.228526 12156 consensus_queue.cc:237] T 00000000000000000000000000000000 P fec2dfb353f443c8b10a94d83571bba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec2dfb353f443c8b10a94d83571bba5" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 35133 } }
I20260928 23:50:00.229203 12151 sys_catalog.cc:564] T 00000000000000000000000000000000 P fec2dfb353f443c8b10a94d83571bba5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:50:00.229622 12157 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec2dfb353f443c8b10a94d83571bba5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fec2dfb353f443c8b10a94d83571bba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec2dfb353f443c8b10a94d83571bba5" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 35133 } } }
I20260928 23:50:00.229818 12157 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec2dfb353f443c8b10a94d83571bba5 [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:00.230064 12158 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec2dfb353f443c8b10a94d83571bba5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fec2dfb353f443c8b10a94d83571bba5. Latest consensus state: current_term: 1 leader_uuid: "fec2dfb353f443c8b10a94d83571bba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec2dfb353f443c8b10a94d83571bba5" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 35133 } } }
I20260928 23:50:00.230159 12158 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec2dfb353f443c8b10a94d83571bba5 [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:00.230366 12162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:50:00.233345 12162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:50:00.238576 12162 catalog_manager.cc:1347] Generated new cluster ID: 92ca62eee37d48a2a25c14c124adc1f1
I20260928 23:50:00.238638 12162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:50:00.251655 12162 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:50:00.252540 12162 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:50:00.262144 12162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fec2dfb353f443c8b10a94d83571bba5: Generated new TSK 0
I20260928 23:50:00.262846 12162 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:50:00.279500 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:00.274806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57204 (local address 127.6.152.62:35133)
0928 23:50:00.274947 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:00.274952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:00.275156 (+ 204us) server_negotiation.cc:407] Connection header received
0928 23:50:00.275340 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:00.275343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:00.275388 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:00.275498 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0928 23:50:00.275753 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:00.276282 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:00.277232 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:00.277452 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:00.277564 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:00.277864 (+ 300us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 23:50:00.279155 (+ 1291us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39183/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 23:50:00.279365 (+ 210us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57204: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39183/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 23:50:00.279567 12154 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57204: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39183/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 23:50:00.281415 12180 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:00.274732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:35133 (local address 127.0.0.1:57204)
0928 23:50:00.275062 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:00.275073 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:00.275197 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:00.275499 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:00.275502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:00.275510 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 23:50:00.275641 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:00.275647 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:00.276395 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:00.276400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:00.277111 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:00.277120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:00.277577 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:00.277581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:00.277672 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:00.277693 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 23:50:00.279323 (+ 1630us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39183/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 23:50:00.281233 (+ 1910us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.152.62:35133: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39183/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":1712,"thread_start_us":103,"threads_started":1}
I20260928 23:50:00.282117 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 12084
E20260928 23:50:00.292507 12181 webserver.cc:533] Webserver: sslize failed:
E20260928 23:50:00.292610 12181 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T23:50:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (292 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T23:50:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:50:00Z Disabled control of system clock
I20260928 23:50:00.330518 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:36651
--webserver_interface=127.6.152.62
--webserver_port=0
--builtin_ntp_servers=127.6.152.20:33783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:36651
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639374546653-6752-0/testchainca.cert with env {}
W20260928 23:50:00.436685 12185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:00.436949 12185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:00.436988 12185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:00.440774 12185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:50:00.440873 12185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:00.440902 12185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:00.440936 12185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:50:00.440953 12185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 23:50:00.440973 12185 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 23:50:00.441000 12185 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 23:50:00.446147 12185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:33783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:36651
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:36651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639374546653-6752-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:00.447084 12185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:00.447964 12185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:00.456125 12190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:00.456195 12185 server_base.cc:1034] running on GCE node
W20260928 23:50:00.456101 12193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:00.456132 12191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:00.456866 12185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:00.457919 12185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:00.459115 12185 hybrid_clock.cc:648] HybridClock initialized: now 1790639400459085 us; error 42 us; skew 500 ppm
I20260928 23:50:00.461015 12185 webserver.cc:460] Webserver started at http://127.6.152.62:44545/ using document root <none> and password file <none>
I20260928 23:50:00.461565 12185 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:00.461654 12185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:00.461872 12185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:00.463820 12185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "ce7b877446ac453cb29edc946a0f959e"
format_stamp: "Formatted at 2026-09-28 23:50:00 on dist-test-slave-p5bw"
server_key: "c2539fcda80062407ec77f144875139e"
server_key_iv: "aa9782e10b9ab1b942293d9a121cdee8"
server_key_version: "encryptionkey@0"
I20260928 23:50:00.464385 12185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "ce7b877446ac453cb29edc946a0f959e"
format_stamp: "Formatted at 2026-09-28 23:50:00 on dist-test-slave-p5bw"
server_key: "c2539fcda80062407ec77f144875139e"
server_key_iv: "aa9782e10b9ab1b942293d9a121cdee8"
server_key_version: "encryptionkey@0"
I20260928 23:50:00.467855 12185 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 23:50:00.470275 12199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:00.471434 12185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:50:00.471573 12185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "ce7b877446ac453cb29edc946a0f959e"
format_stamp: "Formatted at 2026-09-28 23:50:00 on dist-test-slave-p5bw"
server_key: "c2539fcda80062407ec77f144875139e"
server_key_iv: "aa9782e10b9ab1b942293d9a121cdee8"
server_key_version: "encryptionkey@0"
I20260928 23:50:00.471683 12185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:00.482893 12185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:00.483522 12185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:00.483712 12185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:00.490604 12185 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:36651
I20260928 23:50:00.490609 12251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:36651 every 8 connection(s)
I20260928 23:50:00.491550 12185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:50:00.494578 12252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:00.495632 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 12185
I20260928 23:50:00.495760 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:50:00.496088 6752 external_mini_cluster.cc:1428] Setting key e879b5e7822a486a54ed553e625f39b4
I20260928 23:50:00.500609 12252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce7b877446ac453cb29edc946a0f959e: Bootstrap starting.
I20260928 23:50:00.502821 12252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce7b877446ac453cb29edc946a0f959e: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:00.503590 12252 log.cc:824] T 00000000000000000000000000000000 P ce7b877446ac453cb29edc946a0f959e: Log is configured to *not* fsync() on all Append() calls
I20260928 23:50:00.505220 12255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:00.497450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42348 (local address 127.6.152.62:36651)
0928 23:50:00.498076 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:00.498091 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:00.498125 (+ 34us) server_negotiation.cc:407] Connection header received
0928 23:50:00.498730 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:00.498746 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:00.499139 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:00.499471 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:00.499911 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:00.500968 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:00.501828 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:00.502304 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:00.502484 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:00.503374 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:00.503380 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:00.503403 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:50:00.503831 (+ 428us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:00.503890 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:50:00.503910 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:00.504037 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:00.504192 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 23:50:00.504455 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":118,"threads_started":1}
I20260928 23:50:00.505671 12252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce7b877446ac453cb29edc946a0f959e: No bootstrap required, opened a new log
I20260928 23:50:00.508095 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:50:00.509905 12252 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce7b877446ac453cb29edc946a0f959e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce7b877446ac453cb29edc946a0f959e" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36651 } }
I20260928 23:50:00.510149 12252 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce7b877446ac453cb29edc946a0f959e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:00.510238 12252 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce7b877446ac453cb29edc946a0f959e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce7b877446ac453cb29edc946a0f959e, State: Initialized, Role: FOLLOWER
I20260928 23:50:00.510629 12252 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce7b877446ac453cb29edc946a0f959e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce7b877446ac453cb29edc946a0f959e" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36651 } }
I20260928 23:50:00.510772 12252 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce7b877446ac453cb29edc946a0f959e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:50:00.510855 12252 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce7b877446ac453cb29edc946a0f959e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:50:00.510950 12252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce7b877446ac453cb29edc946a0f959e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:00.511813 12252 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce7b877446ac453cb29edc946a0f959e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce7b877446ac453cb29edc946a0f959e" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36651 } }
I20260928 23:50:00.512146 12252 leader_election.cc:302] T 00000000000000000000000000000000 P ce7b877446ac453cb29edc946a0f959e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce7b877446ac453cb29edc946a0f959e; no voters:
I20260928 23:50:00.512430 12252 leader_election.cc:288] T 00000000000000000000000000000000 P ce7b877446ac453cb29edc946a0f959e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:50:00.512550 12257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce7b877446ac453cb29edc946a0f959e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:50:00.512789 12257 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce7b877446ac453cb29edc946a0f959e [term 1 LEADER]: Becoming Leader. State: Replica: ce7b877446ac453cb29edc946a0f959e, State: Running, Role: LEADER
I20260928 23:50:00.513139 12257 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce7b877446ac453cb29edc946a0f959e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce7b877446ac453cb29edc946a0f959e" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36651 } }
I20260928 23:50:00.513628 12252 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce7b877446ac453cb29edc946a0f959e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:50:00.514622 12259 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7b877446ac453cb29edc946a0f959e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce7b877446ac453cb29edc946a0f959e. Latest consensus state: current_term: 1 leader_uuid: "ce7b877446ac453cb29edc946a0f959e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce7b877446ac453cb29edc946a0f959e" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36651 } } }
I20260928 23:50:00.514753 12259 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7b877446ac453cb29edc946a0f959e [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:00.515082 12258 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7b877446ac453cb29edc946a0f959e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce7b877446ac453cb29edc946a0f959e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce7b877446ac453cb29edc946a0f959e" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36651 } } }
I20260928 23:50:00.515178 12258 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7b877446ac453cb29edc946a0f959e [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:00.515905 12266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:50:00.518416 12266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:50:00.523648 12266 catalog_manager.cc:1347] Generated new cluster ID: 7d95296485984cca89a0a8218e478b91
I20260928 23:50:00.523710 12266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:50:00.549526 12266 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:50:00.550794 12266 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:50:00.566262 12266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce7b877446ac453cb29edc946a0f959e: Generated new TSK 0
I20260928 23:50:00.566999 12266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:50:01.063045 12255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:00.580750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42364 (local address 127.6.152.62:36651)
0928 23:50:00.580902 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:00.580907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:00.581059 (+ 152us) server_negotiation.cc:407] Connection header received
0928 23:50:00.581223 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:00.581226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:00.581282 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:00.581414 (+ 132us) server_negotiation.cc:226] Negotiated authn=JWT
0928 23:50:00.581611 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:00.582429 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:00.583275 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:00.583634 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:00.583736 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:00.583997 (+ 261us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 23:50:01.062625 (+478628us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 23:50:01.062920 (+ 295us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42364: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 23:50:01.063161 12255 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42364: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 23:50:01.063308 12281 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:00.580646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36651 (local address 127.0.0.1:42364)
0928 23:50:00.580964 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:00.580979 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:00.581093 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:00.581407 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:00.581409 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:00.581415 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0928 23:50:00.581510 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:00.581514 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:00.582552 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:00.582554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:00.583162 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:00.583168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:00.583749 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:00.583752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:00.583817 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:00.583837 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 23:50:01.062904 (+479067us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 23:50:01.063099 (+ 195us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.152.62:36651: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":97,"threads_started":1}
I20260928 23:50:01.063916 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 12185
2026-09-28T23:50:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (762 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (123 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (120 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:50:01 dist-test-slave-p5bw krb5kdc[12293](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:50:01 dist-test-slave-p5bw krb5kdc[12293](info): set up 2 sockets
Sep 28 23:50:01 dist-test-slave-p5bw krb5kdc[12293](info): commencing operation
krb5kdc: starting...
W20260928 23:50:02.361824 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:50:02 dist-test-slave-p5bw krb5kdc[12293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:50:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:50:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:50:02.484799 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:32993
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:34677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:32993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:50:02.591552 12306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:02.591830 12306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:02.591909 12306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:02.595836 12306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:50:02.595927 12306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:02.595965 12306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:02.596024 12306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:50:02.596191 12306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:50:02.601704 12306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:34677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:32993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:32993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:02.602766 12306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:02.603801 12306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:02.612308 12314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:02.612341 12312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:02.612342 12306 server_base.cc:1034] running on GCE node
W20260928 23:50:02.612371 12311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:02.613204 12306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:02.614312 12306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:02.615494 12306 hybrid_clock.cc:648] HybridClock initialized: now 1790639402615474 us; error 29 us; skew 500 ppm
Sep 28 23:50:02 dist-test-slave-p5bw krb5kdc[12293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639402, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:02.622540 12306 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:50:02.623509 12306 webserver.cc:460] Webserver started at http://127.6.152.62:39967/ using document root <none> and password file <none>
I20260928 23:50:02.624080 12306 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:02.624171 12306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:02.624403 12306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:02.626571 12306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "07d6e22c173e4aa8afc1cf7641da1085"
format_stamp: "Formatted at 2026-09-28 23:50:02 on dist-test-slave-p5bw"
server_key: "da60e82d789b621a8905a17f7d7ed3cf"
server_key_iv: "7d578c4268c3ff20c1a4db6ff228883f"
server_key_version: "encryptionkey@0"
I20260928 23:50:02.627146 12306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "07d6e22c173e4aa8afc1cf7641da1085"
format_stamp: "Formatted at 2026-09-28 23:50:02 on dist-test-slave-p5bw"
server_key: "da60e82d789b621a8905a17f7d7ed3cf"
server_key_iv: "7d578c4268c3ff20c1a4db6ff228883f"
server_key_version: "encryptionkey@0"
I20260928 23:50:02.631148 12306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260928 23:50:02.633605 12321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:02.634799 12306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:50:02.634940 12306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "07d6e22c173e4aa8afc1cf7641da1085"
format_stamp: "Formatted at 2026-09-28 23:50:02 on dist-test-slave-p5bw"
server_key: "da60e82d789b621a8905a17f7d7ed3cf"
server_key_iv: "7d578c4268c3ff20c1a4db6ff228883f"
server_key_version: "encryptionkey@0"
I20260928 23:50:02.635052 12306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:02.648665 12306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:02.651602 12306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:02.651806 12306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:02.658747 12306 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:32993
I20260928 23:50:02.658757 12373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:32993 every 8 connection(s)
I20260928 23:50:02.659698 12306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:50:02.660452 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 12306
I20260928 23:50:02.660590 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:50:02.660931 6752 external_mini_cluster.cc:1428] Setting key f04ac20752b14830a32f8b555754f9e5
I20260928 23:50:02.662788 12374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:02.669440 12374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07d6e22c173e4aa8afc1cf7641da1085: Bootstrap starting.
Sep 28 23:50:02 dist-test-slave-p5bw krb5kdc[12293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:50:02.671787 12374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07d6e22c173e4aa8afc1cf7641da1085: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:02.672494 12374 log.cc:824] T 00000000000000000000000000000000 P 07d6e22c173e4aa8afc1cf7641da1085: Log is configured to *not* fsync() on all Append() calls
I20260928 23:50:02.674404 12374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07d6e22c173e4aa8afc1cf7641da1085: No bootstrap required, opened a new log
I20260928 23:50:02.674962 12377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:02.662217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42708 (local address 127.6.152.62:32993)
0928 23:50:02.662607 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:02.662625 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:02.662672 (+ 47us) server_negotiation.cc:407] Connection header received
0928 23:50:02.663335 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:02.663353 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:02.663670 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:02.663942 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:02.664913 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:02.665928 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:02.667059 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:02.667499 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:02.667611 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:02.670419 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:02.670423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:02.670435 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:02.670463 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:02.672204 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:02.672639 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:02.672647 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:02.672657 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:02.672762 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:02.673085 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:02.673088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:02.673090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:02.673465 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:02.673676 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:02.673939 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 23:50:02.674224 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":146,"threads_started":1}
I20260928 23:50:02.678320 12374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07d6e22c173e4aa8afc1cf7641da1085 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d6e22c173e4aa8afc1cf7641da1085" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 32993 } }
I20260928 23:50:02.678609 12374 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07d6e22c173e4aa8afc1cf7641da1085 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:02.678690 12374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07d6e22c173e4aa8afc1cf7641da1085 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07d6e22c173e4aa8afc1cf7641da1085, State: Initialized, Role: FOLLOWER
I20260928 23:50:02.679100 12374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07d6e22c173e4aa8afc1cf7641da1085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d6e22c173e4aa8afc1cf7641da1085" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 32993 } }
I20260928 23:50:02.679250 12374 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07d6e22c173e4aa8afc1cf7641da1085 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:50:02.679335 12374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07d6e22c173e4aa8afc1cf7641da1085 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:50:02.679431 12374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07d6e22c173e4aa8afc1cf7641da1085 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:02.680536 12374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07d6e22c173e4aa8afc1cf7641da1085 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d6e22c173e4aa8afc1cf7641da1085" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 32993 } }
I20260928 23:50:02.680858 12374 leader_election.cc:302] T 00000000000000000000000000000000 P 07d6e22c173e4aa8afc1cf7641da1085 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07d6e22c173e4aa8afc1cf7641da1085; no voters:
I20260928 23:50:02.681138 12374 leader_election.cc:288] T 00000000000000000000000000000000 P 07d6e22c173e4aa8afc1cf7641da1085 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:50:02.681244 12379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07d6e22c173e4aa8afc1cf7641da1085 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:50:02.681507 12379 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07d6e22c173e4aa8afc1cf7641da1085 [term 1 LEADER]: Becoming Leader. State: Replica: 07d6e22c173e4aa8afc1cf7641da1085, State: Running, Role: LEADER
I20260928 23:50:02.681835 12379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07d6e22c173e4aa8afc1cf7641da1085 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d6e22c173e4aa8afc1cf7641da1085" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 32993 } }
I20260928 23:50:02.682325 12374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07d6e22c173e4aa8afc1cf7641da1085 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:50:02.683490 12381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07d6e22c173e4aa8afc1cf7641da1085 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07d6e22c173e4aa8afc1cf7641da1085. Latest consensus state: current_term: 1 leader_uuid: "07d6e22c173e4aa8afc1cf7641da1085" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d6e22c173e4aa8afc1cf7641da1085" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 32993 } } }
I20260928 23:50:02.683634 12381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07d6e22c173e4aa8afc1cf7641da1085 [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:02.683912 12380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07d6e22c173e4aa8afc1cf7641da1085 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07d6e22c173e4aa8afc1cf7641da1085" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d6e22c173e4aa8afc1cf7641da1085" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 32993 } } }
I20260928 23:50:02.684020 12388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:50:02.684092 12380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07d6e22c173e4aa8afc1cf7641da1085 [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:02.687109 12388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:50:02.691915 12388 catalog_manager.cc:1347] Generated new cluster ID: 2d9478e9a7e84d56bdf344473cdf8028
I20260928 23:50:02.691982 12388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:50:02.704962 12388 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:50:02.705840 12388 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:50:02.723518 12388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07d6e22c173e4aa8afc1cf7641da1085: Generated new TSK 0
I20260928 23:50:02.724251 12388 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
I20260928 23:50:02.755481 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:32993
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:34677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:50:02.862742 12400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:02.863046 12400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:02.863114 12400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:02.867084 12400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:02.867168 12400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:02.867260 12400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:50:02.872359 12400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:34677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:32993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:02.873396 12400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:02.874392 12400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:02.883819 12406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:02.883836 12405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:02.883834 12408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:02.883986 12400 server_base.cc:1034] running on GCE node
I20260928 23:50:02.884702 12400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:02.885349 12400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:02.886512 12400 hybrid_clock.cc:648] HybridClock initialized: now 1790639402886497 us; error 29 us; skew 500 ppm
Sep 28 23:50:02 dist-test-slave-p5bw krb5kdc[12293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639402, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:02.889353 12400 init.cc:377] Logged in from keytab as kudu/127.6.152.1@KRBTEST.COM (short username kudu)
I20260928 23:50:02.890244 12400 webserver.cc:460] Webserver started at http://127.6.152.1:36171/ using document root <none> and password file <none>
I20260928 23:50:02.890887 12400 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:02.890955 12400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:02.891179 12400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:02.893163 12400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "e62afe748ccc49988d7ebc6a2edfb85b"
format_stamp: "Formatted at 2026-09-28 23:50:02 on dist-test-slave-p5bw"
server_key: "83805970ceadc2ce237c2ed183129637"
server_key_iv: "ff39707718d359dee72298a0b8ea6362"
server_key_version: "encryptionkey@0"
I20260928 23:50:02.893702 12400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "e62afe748ccc49988d7ebc6a2edfb85b"
format_stamp: "Formatted at 2026-09-28 23:50:02 on dist-test-slave-p5bw"
server_key: "83805970ceadc2ce237c2ed183129637"
server_key_iv: "ff39707718d359dee72298a0b8ea6362"
server_key_version: "encryptionkey@0"
I20260928 23:50:02.897243 12400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:50:02.899533 12415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:02.900750 12400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:50:02.900913 12400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "e62afe748ccc49988d7ebc6a2edfb85b"
format_stamp: "Formatted at 2026-09-28 23:50:02 on dist-test-slave-p5bw"
server_key: "83805970ceadc2ce237c2ed183129637"
server_key_iv: "ff39707718d359dee72298a0b8ea6362"
server_key_version: "encryptionkey@0"
I20260928 23:50:02.901043 12400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:02.919997 12400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:02.923343 12400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:02.923557 12400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:02.924140 12400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:50:02.924968 12400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:02.925037 12400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:02.925105 12400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:02.925158 12400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:02.933990 12400 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:33701
I20260928 23:50:02.934015 12528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:33701 every 8 connection(s)
I20260928 23:50:02.934846 12400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:50:02.941447 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 12400
I20260928 23:50:02.941546 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
I20260928 23:50:02.941850 6752 external_mini_cluster.cc:1428] Setting key a9aa735ae487e8e4095604fba938bc1d
Sep 28 23:50:02 dist-test-slave-p5bw krb5kdc[12293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639402, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:50:02.948427 12377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:02.936621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:39993 (local address 127.6.152.62:32993)
0928 23:50:02.936753 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:02.936757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:02.937573 (+ 816us) server_negotiation.cc:407] Connection header received
0928 23:50:02.939025 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:02.939028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:02.939086 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:02.939167 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:02.940303 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:02.940913 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:02.942187 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:02.942426 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:02.942549 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:02.944965 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:02.944968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:02.944970 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:02.944993 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:02.946410 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:02.946924 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:02.946927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:02.946929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:02.947016 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:02.947352 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:02.947357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:02.947359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:02.947542 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:02.947660 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:02.948137 (+ 477us) server_negotiation.cc:299] Negotiation successful
0928 23:50:02.948262 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:50:02.949273 12531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:02.937002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:32993 (local address 127.6.152.1:39993)
0928 23:50:02.937416 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:02.937458 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:02.938821 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:02.939333 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:02.939340 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:02.939682 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:02.940147 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:02.940161 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:02.941049 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:02.941052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:02.942067 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:02.942076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:02.942576 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:02.942580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:02.942751 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:02.943343 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:02.943358 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:02.944739 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:02.946547 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:02.946552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:02.946566 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:02.946819 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:02.947109 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:02.947113 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:02.947115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:02.947256 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:02.947646 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:02.947653 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:02.947911 (+ 258us) client_negotiation.cc:769] Sending connection context
0928 23:50:02.948097 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 23:50:02.948401 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.6.152.2@KRBTEST.COM; defaulting to no policy
I20260928 23:50:02.950519 12529 heartbeater.cc:347] Connected to a master server at 127.6.152.62:32993
I20260928 23:50:02.950765 12529 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:02.951243 12529 heartbeater.cc:511] Master 127.6.152.62:32993 requested a full tablet report, sending...
I20260928 23:50:02.952980 12338 ts_manager.cc:194] Registered new tserver with Master: e62afe748ccc49988d7ebc6a2edfb85b (127.6.152.1:33701)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.2@KRBTEST.COM" created.
I20260928 23:50:02.954734 12338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.1@KRBTEST.COM'} at 127.6.152.1:39993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
I20260928 23:50:02.972750 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:32993
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:34677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:50:03.082255 12534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:03.082504 12534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:03.082549 12534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:03.086637 12534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:03.086690 12534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:03.086747 12534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:50:03.092034 12534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:34677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:32993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:03.093027 12534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:03.093963 12534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:03.103931 12540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:03.104142 12534 server_base.cc:1034] running on GCE node
W20260928 23:50:03.103946 12542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:03.103964 12539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:03.104632 12534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:03.105413 12534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:03.106608 12534 hybrid_clock.cc:648] HybridClock initialized: now 1790639403106590 us; error 25 us; skew 500 ppm
Sep 28 23:50:03 dist-test-slave-p5bw krb5kdc[12293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639403, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:03.109344 12534 init.cc:377] Logged in from keytab as kudu/127.6.152.2@KRBTEST.COM (short username kudu)
I20260928 23:50:03.110203 12534 webserver.cc:460] Webserver started at http://127.6.152.2:37967/ using document root <none> and password file <none>
I20260928 23:50:03.110749 12534 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:03.110837 12534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:03.111068 12534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:03.113106 12534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "23a2a40b09fd4d7493e29b2ad1e27b16"
format_stamp: "Formatted at 2026-09-28 23:50:03 on dist-test-slave-p5bw"
server_key: "c95368157217a523c18495e119a1df90"
server_key_iv: "438131efbd31cbfc2cedb02a5743f90e"
server_key_version: "encryptionkey@0"
I20260928 23:50:03.113653 12534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "23a2a40b09fd4d7493e29b2ad1e27b16"
format_stamp: "Formatted at 2026-09-28 23:50:03 on dist-test-slave-p5bw"
server_key: "c95368157217a523c18495e119a1df90"
server_key_iv: "438131efbd31cbfc2cedb02a5743f90e"
server_key_version: "encryptionkey@0"
I20260928 23:50:03.117204 12534 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:50:03.119491 12549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:03.120680 12534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:50:03.120847 12534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "23a2a40b09fd4d7493e29b2ad1e27b16"
format_stamp: "Formatted at 2026-09-28 23:50:03 on dist-test-slave-p5bw"
server_key: "c95368157217a523c18495e119a1df90"
server_key_iv: "438131efbd31cbfc2cedb02a5743f90e"
server_key_version: "encryptionkey@0"
I20260928 23:50:03.121001 12534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:03.135142 12534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:03.137923 12534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:03.138129 12534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:03.138648 12534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:50:03.139441 12534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:03.139485 12534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:03.139523 12534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:03.139537 12534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:03.147957 12534 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:46637
I20260928 23:50:03.147979 12662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:46637 every 8 connection(s)
I20260928 23:50:03.148803 12534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
Sep 28 23:50:03 dist-test-slave-p5bw krb5kdc[12293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639403, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:50:03.159614 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 12534
I20260928 23:50:03.159751 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:50:03.160060 6752 external_mini_cluster.cc:1428] Setting key e379423f583d8f09ebaebfcb338bf5ba
I20260928 23:50:03.164037 12377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:03.150641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:53663 (local address 127.6.152.62:32993)
0928 23:50:03.150811 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:03.150816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:03.151632 (+ 816us) server_negotiation.cc:407] Connection header received
0928 23:50:03.153013 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:03.153015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:03.153069 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:03.153146 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:03.154307 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:03.154826 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:03.156132 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:03.156453 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:03.156565 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:03.159932 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:03.159936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:03.159939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:03.159969 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:03.161786 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:03.162317 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:03.162323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:03.162328 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:03.162422 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:03.162756 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:03.162762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:03.162766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:03.163036 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:03.163148 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:03.163688 (+ 540us) server_negotiation.cc:299] Negotiation successful
0928 23:50:03.163829 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 23:50:03.164969 12665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:03.151042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:32993 (local address 127.6.152.2:53663)
0928 23:50:03.151474 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:03.151515 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:03.152758 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:03.153339 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:03.153346 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:03.153696 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:03.154179 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:03.154192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:03.154974 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:03.154976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:03.155907 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:03.155922 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:03.156665 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:03.156672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:03.156972 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:03.157767 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:03.157790 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:03.159764 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:03.161908 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:03.161913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:03.161926 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:03.162209 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:03.162519 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:03.162521 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:03.162523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:03.162661 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:03.163149 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:03.163154 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:03.163444 (+ 290us) client_negotiation.cc:769] Sending connection context
0928 23:50:03.163624 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 23:50:03.163887 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":100,"threads_started":1}
I20260928 23:50:03.166213 12663 heartbeater.cc:347] Connected to a master server at 127.6.152.62:32993
I20260928 23:50:03.166466 12663 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:03.166981 12663 heartbeater.cc:511] Master 127.6.152.62:32993 requested a full tablet report, sending...
I20260928 23:50:03.168093 12338 ts_manager.cc:194] Registered new tserver with Master: 23a2a40b09fd4d7493e29b2ad1e27b16 (127.6.152.2:46637)
I20260928 23:50:03.168718 12338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.2@KRBTEST.COM'} at 127.6.152.2:53663
WARNING: no policy specified for kudu/127.6.152.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
I20260928 23:50:03.187770 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:32993
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:34677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:50:03.295521 12668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:03.295770 12668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:03.295811 12668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:03.299669 12668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:03.299716 12668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:03.299772 12668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:50:03.305032 12668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:34677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:32993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:03.306063 12668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:03.307080 12668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:03.316442 12674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:03.316442 12676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:03.316457 12673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:03.316613 12668 server_base.cc:1034] running on GCE node
I20260928 23:50:03.317178 12668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:03.317785 12668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:03.318941 12668 hybrid_clock.cc:648] HybridClock initialized: now 1790639403318924 us; error 28 us; skew 500 ppm
Sep 28 23:50:03 dist-test-slave-p5bw krb5kdc[12293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639403, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:03.321559 12668 init.cc:377] Logged in from keytab as kudu/127.6.152.3@KRBTEST.COM (short username kudu)
I20260928 23:50:03.322397 12668 webserver.cc:460] Webserver started at http://127.6.152.3:46795/ using document root <none> and password file <none>
I20260928 23:50:03.322906 12668 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:03.322965 12668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:03.323191 12668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:03.325256 12668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "e8402f5bced54672929fb0f315211213"
format_stamp: "Formatted at 2026-09-28 23:50:03 on dist-test-slave-p5bw"
server_key: "663734cd21f2fdd938fd411acb599983"
server_key_iv: "4bf2500d8508a47bc106018c6c03b438"
server_key_version: "encryptionkey@0"
I20260928 23:50:03.325762 12668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "e8402f5bced54672929fb0f315211213"
format_stamp: "Formatted at 2026-09-28 23:50:03 on dist-test-slave-p5bw"
server_key: "663734cd21f2fdd938fd411acb599983"
server_key_iv: "4bf2500d8508a47bc106018c6c03b438"
server_key_version: "encryptionkey@0"
I20260928 23:50:03.329313 12668 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 23:50:03.331651 12683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:03.332959 12668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:50:03.333050 12668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "e8402f5bced54672929fb0f315211213"
format_stamp: "Formatted at 2026-09-28 23:50:03 on dist-test-slave-p5bw"
server_key: "663734cd21f2fdd938fd411acb599983"
server_key_iv: "4bf2500d8508a47bc106018c6c03b438"
server_key_version: "encryptionkey@0"
I20260928 23:50:03.333168 12668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:03.352918 12668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:03.355775 12668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:03.355927 12668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:03.356456 12668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:50:03.357291 12668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:03.357339 12668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:03.357378 12668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:03.357393 12668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:03.366183 12668 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:38843
I20260928 23:50:03.366214 12796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:38843 every 8 connection(s)
I20260928 23:50:03.367075 12668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:50:03.373972 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 12668
I20260928 23:50:03.374070 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:50:03.374367 6752 external_mini_cluster.cc:1428] Setting key 4c1d1ee70bd8d7f312d76b30e173b3a9
Sep 28 23:50:03 dist-test-slave-p5bw krb5kdc[12293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639403, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:50:03.380554 12377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:03.368919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:58213 (local address 127.6.152.62:32993)
0928 23:50:03.369068 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:03.369071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:03.369872 (+ 801us) server_negotiation.cc:407] Connection header received
0928 23:50:03.371308 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:03.371311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:03.371367 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:03.371457 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:03.372596 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:03.373217 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:03.374557 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:03.374775 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:03.374909 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:03.377294 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:03.377298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:03.377307 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:03.377332 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:03.378586 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:03.379104 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:03.379108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:03.379110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:03.379189 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:03.379536 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:03.379541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:03.379543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:03.379720 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:03.379827 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:03.380283 (+ 456us) server_negotiation.cc:299] Negotiation successful
0928 23:50:03.380399 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 23:50:03.381384 12799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:03.369279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:32993 (local address 127.6.152.3:58213)
0928 23:50:03.369714 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:03.369756 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:03.371082 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:03.371633 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:03.371640 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:03.371988 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:03.372453 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:03.372466 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:03.373339 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:03.373344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:03.374365 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:03.374382 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:03.374924 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:03.374928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:03.375134 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:03.375727 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:03.375742 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:03.377138 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:03.378713 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:03.378719 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:03.378733 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:03.378996 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:03.379281 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:03.379286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:03.379288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:03.379439 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:03.379825 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:03.379832 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:03.380060 (+ 228us) client_negotiation.cc:769] Sending connection context
0928 23:50:03.380242 (+ 182us) client_negotiation.cc:240] Negotiation successful
0928 23:50:03.380551 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":96,"threads_started":1}
I20260928 23:50:03.382459 12797 heartbeater.cc:347] Connected to a master server at 127.6.152.62:32993
I20260928 23:50:03.382663 12797 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:03.383168 12797 heartbeater.cc:511] Master 127.6.152.62:32993 requested a full tablet report, sending...
I20260928 23:50:03.384090 12338 ts_manager.cc:194] Registered new tserver with Master: e8402f5bced54672929fb0f315211213 (127.6.152.3:38843)
I20260928 23:50:03.384641 12338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.3@KRBTEST.COM'} at 127.6.152.3:58213
I20260928 23:50:03.388023 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/test-admin.keytab.
Sep 28 23:50:03 dist-test-slave-p5bw krb5kdc[12293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:03.405833 6752 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 23:50:03.408514 12804 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:03.407184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49012 (local address 127.6.152.1:33701)
0928 23:50:03.407473 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:03.407487 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:03.407534 (+ 47us) server_negotiation.cc:407] Connection header received
0928 23:50:03.407585 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:03.407591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:03.407777 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:03.407887 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:03.408332 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49012: BlockingRecv error: recv got EOF from 127.0.0.1:49012 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":92,"threads_started":1}
W20260928 23:50:03.408550 6752 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/krb5cc' of type FILE
Sep 28 23:50:03 dist-test-slave-p5bw krb5kdc[12293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:03.409969 6752 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 23:50:03 dist-test-slave-p5bw krb5kdc[12293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:03.410889 6752 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 23:50:03 dist-test-slave-p5bw krb5kdc[12293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.1@KRBTEST.COM
I20260928 23:50:03.420320 12804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:03.411813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49022 (local address 127.6.152.1:33701)
0928 23:50:03.411938 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:03.411943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:03.412061 (+ 118us) server_negotiation.cc:407] Connection header received
0928 23:50:03.412175 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:03.412178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:03.412218 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:03.412288 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:03.412771 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:03.413625 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:03.414461 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:03.414830 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:03.414923 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:03.417237 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:03.417245 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:03.417258 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:03.417281 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:03.418781 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:03.419162 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:03.419167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:03.419168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:03.419245 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:03.419464 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:03.419466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:03.419467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:03.419686 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:03.419830 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:03.419992 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 23:50:03.420172 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 23:50:03.424403 12804 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:03.423228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49026 (local address 127.6.152.1:33701)
0928 23:50:03.423360 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:03.423364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:03.423466 (+ 102us) server_negotiation.cc:407] Connection header received
0928 23:50:03.423629 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:03.423635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:03.423675 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:03.423757 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:03.424283 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49026: BlockingRecv error: recv got EOF from 127.0.0.1:49026 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 23:50:03.424641 6752 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/krb5cc' of type FILE
Sep 28 23:50:03 dist-test-slave-p5bw krb5kdc[12293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:03.425771 6752 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 23:50:03 dist-test-slave-p5bw krb5kdc[12293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:03.426647 6752 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 23:50:03 dist-test-slave-p5bw krb5kdc[12293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.1@KRBTEST.COM
I20260928 23:50:03.434528 12804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:03.427385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49030 (local address 127.6.152.1:33701)
0928 23:50:03.427498 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:03.427504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:03.427612 (+ 108us) server_negotiation.cc:407] Connection header received
0928 23:50:03.427695 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:03.427697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:03.427729 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:03.427790 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:03.428244 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:03.428737 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:03.429514 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:03.429733 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:03.429809 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:03.431734 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:03.431737 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:03.431739 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:03.431759 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:03.433214 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:03.433556 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:03.433559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:03.433560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:03.433627 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:03.433839 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:03.433841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:03.433842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:03.433980 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:03.434083 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:03.434247 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 23:50:03.434359 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 23:50:03.436864 12804 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:03.436029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49038 (local address 127.6.152.1:33701)
0928 23:50:03.436158 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:03.436162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:03.436231 (+ 69us) server_negotiation.cc:407] Connection header received
0928 23:50:03.436300 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:03.436303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:03.436342 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:03.436419 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:03.436738 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49038: BlockingRecv error: recv got EOF from 127.0.0.1:49038 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 23:50:03.436914 6752 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639374546653-6752-0/krb5kdc/krb5cc' of type FILE
Sep 28 23:50:03 dist-test-slave-p5bw krb5kdc[12293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:03.437932 6752 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 23:50:03 dist-test-slave-p5bw krb5kdc[12293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:03.438827 6752 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 23:50:03 dist-test-slave-p5bw krb5kdc[12293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.1@KRBTEST.COM
I20260928 23:50:03.446842 12804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:03.439633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49042 (local address 127.6.152.1:33701)
0928 23:50:03.439760 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:03.439763 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:03.439776 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:50:03.439863 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:03.439865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:03.439901 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:03.439959 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:03.440384 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:03.440945 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:03.441691 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:03.441937 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:03.442031 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:03.444082 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:03.444085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:03.444087 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:03.444108 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:03.445423 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:03.445798 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:03.445801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:03.445802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:03.445878 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:03.446142 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:03.446144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:03.446145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:03.446289 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:03.446382 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:03.446549 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 23:50:03.446671 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 23:50:03.447824 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 12400
I20260928 23:50:03.453751 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 12534
I20260928 23:50:03.458822 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 12668
I20260928 23:50:03.463874 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 12306
2026-09-28T23:50:03Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2156 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:50:03 dist-test-slave-p5bw krb5kdc[12817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:50:03 dist-test-slave-p5bw krb5kdc[12817](info): set up 2 sockets
Sep 28 23:50:03 dist-test-slave-p5bw krb5kdc[12817](info): commencing operation
krb5kdc: starting...
W20260928 23:50:04.525804 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:50:04 dist-test-slave-p5bw krb5kdc[12817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:50:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:50:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.62.keytab.
Entry for principal oryx/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.62.keytab.
I20260928 23:50:04.649714 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:36193
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.62.keytab
--principal=oryx/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:38547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:36193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:50:04.756289 12830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:04.756574 12830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:04.756644 12830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:04.760547 12830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:50:04.760637 12830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:04.760676 12830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:04.760733 12830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:50:04.760773 12830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:50:04.765882 12830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:38547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:36193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.62.keytab
--principal=oryx/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:36193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:04.766892 12830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:04.767870 12830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:04.776456 12835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:04.776412 12836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:04.776515 12838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:04.776655 12830 server_base.cc:1034] running on GCE node
I20260928 23:50:04.777365 12830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:04.778424 12830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:04.779639 12830 hybrid_clock.cc:648] HybridClock initialized: now 1790639404779623 us; error 27 us; skew 500 ppm
Sep 28 23:50:04 dist-test-slave-p5bw krb5kdc[12817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639404, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:04.782377 12830 init.cc:377] Logged in from keytab as oryx/127.6.152.62@KRBTEST.COM (short username oryx)
I20260928 23:50:04.783267 12830 webserver.cc:460] Webserver started at http://127.6.152.62:45633/ using document root <none> and password file <none>
I20260928 23:50:04.783818 12830 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:04.783916 12830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:04.784144 12830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:04.786208 12830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "8b20d904b67749e9a3276bfdd467c7be"
format_stamp: "Formatted at 2026-09-28 23:50:04 on dist-test-slave-p5bw"
server_key: "4a066b467181ab35646712a98b7389d8"
server_key_iv: "0bab8fb0f6c8ce83b3d315c21c2a7af4"
server_key_version: "encryptionkey@0"
I20260928 23:50:04.786767 12830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "8b20d904b67749e9a3276bfdd467c7be"
format_stamp: "Formatted at 2026-09-28 23:50:04 on dist-test-slave-p5bw"
server_key: "4a066b467181ab35646712a98b7389d8"
server_key_iv: "0bab8fb0f6c8ce83b3d315c21c2a7af4"
server_key_version: "encryptionkey@0"
I20260928 23:50:04.790488 12830 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260928 23:50:04.792783 12845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:04.794067 12830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:50:04.794202 12830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "8b20d904b67749e9a3276bfdd467c7be"
format_stamp: "Formatted at 2026-09-28 23:50:04 on dist-test-slave-p5bw"
server_key: "4a066b467181ab35646712a98b7389d8"
server_key_iv: "0bab8fb0f6c8ce83b3d315c21c2a7af4"
server_key_version: "encryptionkey@0"
I20260928 23:50:04.794313 12830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:04.815477 12830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:04.818786 12830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:04.818994 12830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:04.826174 12830 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:36193
I20260928 23:50:04.826467 12897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:36193 every 8 connection(s)
I20260928 23:50:04.827167 12830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:50:04.829910 12898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:04.835224 12898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be: Bootstrap starting.
I20260928 23:50:04.835494 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 12830
I20260928 23:50:04.835595 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:50:04.835803 6752 external_mini_cluster.cc:1428] Setting key 602c416c5bab811f4e4d3883a159a3f2
I20260928 23:50:04.837671 12898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:04.838496 12898 log.cc:824] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be: Log is configured to *not* fsync() on all Append() calls
I20260928 23:50:04.840633 12898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be: No bootstrap required, opened a new log
I20260928 23:50:04.844560 12898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } }
I20260928 23:50:04.844851 12898 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:04.844913 12898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b20d904b67749e9a3276bfdd467c7be, State: Initialized, Role: FOLLOWER
Sep 28 23:50:04 dist-test-slave-p5bw krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM
I20260928 23:50:04.845395 12898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } }
I20260928 23:50:04.845566 12898 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:50:04.845662 12898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:50:04.845763 12898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:04.846753 12898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } }
I20260928 23:50:04.847116 12898 leader_election.cc:302] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b20d904b67749e9a3276bfdd467c7be; no voters:
I20260928 23:50:04.847443 12898 leader_election.cc:288] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:50:04.847520 12903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:50:04.847723 12903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [term 1 LEADER]: Becoming Leader. State: Replica: 8b20d904b67749e9a3276bfdd467c7be, State: Running, Role: LEADER
I20260928 23:50:04.848047 12903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } }
I20260928 23:50:04.848795 12898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:50:04.851276 12905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b20d904b67749e9a3276bfdd467c7be. Latest consensus state: current_term: 1 leader_uuid: "8b20d904b67749e9a3276bfdd467c7be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } }
I20260928 23:50:04.851292 12904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b20d904b67749e9a3276bfdd467c7be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } }
I20260928 23:50:04.851858 12905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:04.851864 12904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:04.851439 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:04.837106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53078 (local address 127.6.152.62:36193)
0928 23:50:04.837507 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:04.837527 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:04.837571 (+ 44us) server_negotiation.cc:407] Connection header received
0928 23:50:04.838430 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:04.838452 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:04.838857 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:04.839150 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:04.839979 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:04.841437 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:04.842292 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:04.843015 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:04.843192 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:04.846011 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:04.846017 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:04.846039 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:04.846065 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:04.847998 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:04.848476 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:04.848481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:04.848487 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:04.848625 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:04.848976 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:04.848980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:04.848981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:04.849487 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:04.849700 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:04.850155 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 23:50:04.850439 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":107,"threads_started":1}
W20260928 23:50:04.853706 12918 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:50:04.853789 12918 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:50:04.853858 12919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:50:04.854609 12919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:50:04.859505 12919 catalog_manager.cc:1347] Generated new cluster ID: cef39238cd1d47c9b5fe97b10e1b2ace
I20260928 23:50:04.859561 12919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:50:04.880112 12919 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:50:04.881120 12919 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:50:04.885298 12919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be: Generated new TSK 0
I20260928 23:50:04.886132 12919 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.1.keytab.
Entry for principal oryx/127.6.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.1.keytab.
I20260928 23:50:04.921530 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:36193
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.1.keytab
--principal=oryx/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:38547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:50:05.030930 12924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:05.031226 12924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:05.031302 12924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:05.035292 12924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:05.035383 12924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:05.035475 12924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:50:05.040647 12924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:38547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.1.keytab
--principal=oryx/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:36193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:05.041704 12924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:05.042734 12924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:05.052227 12929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:05.052287 12924 server_base.cc:1034] running on GCE node
W20260928 23:50:05.052194 12930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:05.052214 12932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:05.052772 12924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:05.053419 12924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:05.054608 12924 hybrid_clock.cc:648] HybridClock initialized: now 1790639405054588 us; error 36 us; skew 500 ppm
Sep 28 23:50:05 dist-test-slave-p5bw krb5kdc[12817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639405, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:05.057296 12924 init.cc:377] Logged in from keytab as oryx/127.6.152.1@KRBTEST.COM (short username oryx)
I20260928 23:50:05.058151 12924 webserver.cc:460] Webserver started at http://127.6.152.1:36617/ using document root <none> and password file <none>
I20260928 23:50:05.058708 12924 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:05.058765 12924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:05.058995 12924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:05.061092 12924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "af5a3b2b50fa4554a12490c58ccbb825"
format_stamp: "Formatted at 2026-09-28 23:50:05 on dist-test-slave-p5bw"
server_key: "33f0f2275a25a9663354ffdbe899003a"
server_key_iv: "39dd5928d35f9b65630fc4baf760314f"
server_key_version: "encryptionkey@0"
I20260928 23:50:05.061668 12924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "af5a3b2b50fa4554a12490c58ccbb825"
format_stamp: "Formatted at 2026-09-28 23:50:05 on dist-test-slave-p5bw"
server_key: "33f0f2275a25a9663354ffdbe899003a"
server_key_iv: "39dd5928d35f9b65630fc4baf760314f"
server_key_version: "encryptionkey@0"
I20260928 23:50:05.065765 12924 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:50:05.068113 12939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:05.069456 12924 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:50:05.069595 12924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "af5a3b2b50fa4554a12490c58ccbb825"
format_stamp: "Formatted at 2026-09-28 23:50:05 on dist-test-slave-p5bw"
server_key: "33f0f2275a25a9663354ffdbe899003a"
server_key_iv: "39dd5928d35f9b65630fc4baf760314f"
server_key_version: "encryptionkey@0"
I20260928 23:50:05.069705 12924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:05.086700 12924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:05.089704 12924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:05.089916 12924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:05.090494 12924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:50:05.091310 12924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:05.091379 12924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:05.091449 12924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:05.091492 12924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:05.100780 12924 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:38171
I20260928 23:50:05.100800 13053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:38171 every 8 connection(s)
I20260928 23:50:05.101678 12924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:50:05.107847 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 12924
I20260928 23:50:05.107944 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
I20260928 23:50:05.108234 6752 external_mini_cluster.cc:1428] Setting key 19dad80d700f834c197ed5f1c2b32a10
Sep 28 23:50:05 dist-test-slave-p5bw krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639405, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM
I20260928 23:50:05.116392 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:05.103509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:38881 (local address 127.6.152.62:36193)
0928 23:50:05.103658 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:05.103662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:05.104604 (+ 942us) server_negotiation.cc:407] Connection header received
0928 23:50:05.105639 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:05.105641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:05.105688 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:05.105760 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:05.106929 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.107444 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.108843 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.109137 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.109258 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:05.112114 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:05.112119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:05.112122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:05.112150 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:05.113849 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:05.114642 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:05.114645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:05.114646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:05.114718 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:05.115155 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:05.115157 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:05.115158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:05.115321 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:05.115393 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:05.116137 (+ 744us) server_negotiation.cc:299] Negotiation successful
0928 23:50:05.116258 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 23:50:05.117658 13056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:05.103798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.6.152.1:38881)
0928 23:50:05.104471 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:05.104516 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:05.105415 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:05.105908 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:05.105915 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:05.106280 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:05.106792 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:05.106806 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.107552 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.107555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:05.108700 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:05.108710 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.109292 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.109296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:05.109516 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:05.110182 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:05.110197 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:05.111941 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:05.114079 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:05.114089 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:05.114110 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:05.114537 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:05.114818 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:05.114824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:05.114830 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:05.115039 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:05.115434 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:05.115443 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:05.115811 (+ 368us) client_negotiation.cc:769] Sending connection context
0928 23:50:05.116068 (+ 257us) client_negotiation.cc:240] Negotiation successful
0928 23:50:05.116466 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":127,"threads_started":1}
WARNING: no policy specified for oryx/127.6.152.2@KRBTEST.COM; defaulting to no policy
I20260928 23:50:05.119071 13054 heartbeater.cc:347] Connected to a master server at 127.6.152.62:36193
I20260928 23:50:05.119336 13054 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:05.119802 13054 heartbeater.cc:511] Master 127.6.152.62:36193 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.152.2@KRBTEST.COM" created.
I20260928 23:50:05.121649 12862 ts_manager.cc:194] Registered new tserver with Master: af5a3b2b50fa4554a12490c58ccbb825 (127.6.152.1:38171)
I20260928 23:50:05.122972 12862 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.152.1@KRBTEST.COM'} at 127.6.152.1:38881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.152.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.2.keytab.
Entry for principal oryx/127.6.152.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.2.keytab.
I20260928 23:50:05.136632 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:36193
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.2.keytab
--principal=oryx/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:38547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:50:05.245405 13059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:05.245684 13059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:05.245755 13059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:05.250088 13059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:05.250156 13059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:05.250249 13059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:50:05.255298 13059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:38547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.2.keytab
--principal=oryx/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:36193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:05.256264 13059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:05.257287 13059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:05.266571 13064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:05.266606 13065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:05.266598 13067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:05.266804 13059 server_base.cc:1034] running on GCE node
I20260928 23:50:05.267199 13059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:05.267817 13059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:05.269026 13059 hybrid_clock.cc:648] HybridClock initialized: now 1790639405269015 us; error 28 us; skew 500 ppm
Sep 28 23:50:05 dist-test-slave-p5bw krb5kdc[12817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639405, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:05.271728 13059 init.cc:377] Logged in from keytab as oryx/127.6.152.2@KRBTEST.COM (short username oryx)
I20260928 23:50:05.272569 13059 webserver.cc:460] Webserver started at http://127.6.152.2:44809/ using document root <none> and password file <none>
I20260928 23:50:05.273208 13059 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:05.273301 13059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:05.273573 13059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:05.275573 13059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "90b865839cc84946afb5399f87beee52"
format_stamp: "Formatted at 2026-09-28 23:50:05 on dist-test-slave-p5bw"
server_key: "975d731c0fa135483466bc0932380d62"
server_key_iv: "7302b6613e53fb1be3a135eb0ef62ae8"
server_key_version: "encryptionkey@0"
I20260928 23:50:05.276125 13059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "90b865839cc84946afb5399f87beee52"
format_stamp: "Formatted at 2026-09-28 23:50:05 on dist-test-slave-p5bw"
server_key: "975d731c0fa135483466bc0932380d62"
server_key_iv: "7302b6613e53fb1be3a135eb0ef62ae8"
server_key_version: "encryptionkey@0"
I20260928 23:50:05.279796 13059 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 23:50:05.282269 13075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:05.283610 13059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:50:05.283766 13059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "90b865839cc84946afb5399f87beee52"
format_stamp: "Formatted at 2026-09-28 23:50:05 on dist-test-slave-p5bw"
server_key: "975d731c0fa135483466bc0932380d62"
server_key_iv: "7302b6613e53fb1be3a135eb0ef62ae8"
server_key_version: "encryptionkey@0"
I20260928 23:50:05.283885 13059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:05.301692 13059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:05.304659 13059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:05.304917 13059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:05.305500 13059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:50:05.306316 13059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:05.306394 13059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:05.306465 13059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:05.306509 13059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:05.315164 13059 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:46775
I20260928 23:50:05.315191 13189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:46775 every 8 connection(s)
I20260928 23:50:05.316030 13059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:50:05.322890 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 13059
I20260928 23:50:05.323036 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:50:05.323319 6752 external_mini_cluster.cc:1428] Setting key bd775936258b1f621e4c962318122748
Sep 28 23:50:05 dist-test-slave-p5bw krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639405, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM
I20260928 23:50:05.329883 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:05.317758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:58319 (local address 127.6.152.62:36193)
0928 23:50:05.317901 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:05.317905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:05.318834 (+ 929us) server_negotiation.cc:407] Connection header received
0928 23:50:05.319728 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:05.319731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:05.319790 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:05.319900 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:05.321065 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.321601 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.322711 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.322952 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.323065 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:05.325698 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:05.325708 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:05.325711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:05.325733 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:05.327500 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:05.328031 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:05.328060 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:05.328065 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:05.328158 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:05.328540 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:05.328546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:05.328551 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:05.328829 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:05.328940 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:05.329500 (+ 560us) server_negotiation.cc:299] Negotiation successful
0928 23:50:05.329637 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:50:05.330529 13192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:05.318075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.6.152.2:58319)
0928 23:50:05.318681 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:05.318720 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:05.319522 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:05.320080 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:05.320087 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:05.320430 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:05.320909 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:05.320923 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.321725 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.321729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:05.322587 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:05.322596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.323101 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.323106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:05.323269 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:05.323915 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:05.323934 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:05.325541 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:05.327623 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:05.327627 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:05.327639 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:05.327892 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:05.328272 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:05.328274 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:05.328276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:05.328426 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:05.328993 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:05.329001 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:05.329251 (+ 250us) client_negotiation.cc:769] Sending connection context
0928 23:50:05.329410 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 23:50:05.329656 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":100,"threads_started":1}
I20260928 23:50:05.331696 13190 heartbeater.cc:347] Connected to a master server at 127.6.152.62:36193
I20260928 23:50:05.331944 13190 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:05.332532 13190 heartbeater.cc:511] Master 127.6.152.62:36193 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.6.152.3@KRBTEST.COM; defaulting to no policy
I20260928 23:50:05.333634 12862 ts_manager.cc:194] Registered new tserver with Master: 90b865839cc84946afb5399f87beee52 (127.6.152.2:46775)
I20260928 23:50:05.334190 12862 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.152.2@KRBTEST.COM'} at 127.6.152.2:58319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.152.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.152.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.3.keytab.
Entry for principal oryx/127.6.152.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.3.keytab.
I20260928 23:50:05.351531 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:36193
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.3.keytab
--principal=oryx/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:38547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:50:05.460108 13195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:05.460353 13195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:05.460397 13195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:05.464452 13195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:05.464509 13195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:05.464601 13195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:50:05.469762 13195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:38547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.3.keytab
--principal=oryx/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:36193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:05.470770 13195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:05.471810 13195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:05.481233 13200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:05.481208 13203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:05.481297 13201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:05.481592 13195 server_base.cc:1034] running on GCE node
I20260928 23:50:05.481981 13195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:05.482589 13195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:05.483809 13195 hybrid_clock.cc:648] HybridClock initialized: now 1790639405483782 us; error 46 us; skew 500 ppm
Sep 28 23:50:05 dist-test-slave-p5bw krb5kdc[12817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639405, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:05.486886 13195 init.cc:377] Logged in from keytab as oryx/127.6.152.3@KRBTEST.COM (short username oryx)
I20260928 23:50:05.487746 13195 webserver.cc:460] Webserver started at http://127.6.152.3:36631/ using document root <none> and password file <none>
I20260928 23:50:05.488368 13195 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:05.488458 13195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:05.488679 13195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:05.490741 13195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "24026d9eea29427bb69abcf470ecf1da"
format_stamp: "Formatted at 2026-09-28 23:50:05 on dist-test-slave-p5bw"
server_key: "a328a4cc236430af75561d1be529cd85"
server_key_iv: "c88b0fc4e4388bd96850b5e1d7febcb5"
server_key_version: "encryptionkey@0"
I20260928 23:50:05.491298 13195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "24026d9eea29427bb69abcf470ecf1da"
format_stamp: "Formatted at 2026-09-28 23:50:05 on dist-test-slave-p5bw"
server_key: "a328a4cc236430af75561d1be529cd85"
server_key_iv: "c88b0fc4e4388bd96850b5e1d7febcb5"
server_key_version: "encryptionkey@0"
I20260928 23:50:05.494911 13195 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 23:50:05.497443 13210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:05.498737 13195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:50:05.498871 13195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "24026d9eea29427bb69abcf470ecf1da"
format_stamp: "Formatted at 2026-09-28 23:50:05 on dist-test-slave-p5bw"
server_key: "a328a4cc236430af75561d1be529cd85"
server_key_iv: "c88b0fc4e4388bd96850b5e1d7febcb5"
server_key_version: "encryptionkey@0"
I20260928 23:50:05.498989 13195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:05.543365 13195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:05.546309 13195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:05.546530 13195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:05.547123 13195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:50:05.547976 13195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:05.548054 13195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:05.548149 13195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:05.548198 13195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:05.556777 13195 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:33437
I20260928 23:50:05.556818 13324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:33437 every 8 connection(s)
I20260928 23:50:05.557919 13195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
Sep 28 23:50:05 dist-test-slave-p5bw krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639405, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM
I20260928 23:50:05.567867 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 13195
I20260928 23:50:05.568101 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:50:05.568420 6752 external_mini_cluster.cc:1428] Setting key 89028ee6094e1a855f7c3731cf03e7af
I20260928 23:50:05.571799 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:05.559606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:36921 (local address 127.6.152.62:36193)
0928 23:50:05.559752 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:05.559757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:05.560939 (+ 1182us) server_negotiation.cc:407] Connection header received
0928 23:50:05.561849 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:05.561852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:05.561896 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:05.561976 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:05.563205 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.563723 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.565000 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.565208 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.565287 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:05.567858 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:05.567861 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:05.567863 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:05.567886 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:05.569804 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:05.570278 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:05.570282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:05.570284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:05.570380 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:05.570793 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:05.570796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:05.570797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:05.571002 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:05.571093 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:05.571585 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 23:50:05.571683 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:50:05.572645 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:05.560030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.6.152.3:36921)
0928 23:50:05.560734 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:05.560776 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:05.561650 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:05.562149 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:05.562156 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:05.562534 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:05.563072 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:05.563088 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.563873 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.563876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:05.564881 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:05.564893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.565359 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.565363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:05.565505 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:05.566141 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:05.566157 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:05.567689 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:05.569908 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:05.569914 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:05.569929 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:05.570185 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:05.570478 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:05.570480 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:05.570482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:05.570674 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:05.571100 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:05.571105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:05.571337 (+ 232us) client_negotiation.cc:769] Sending connection context
0928 23:50:05.571493 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 23:50:05.571725 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":108,"threads_started":1}
I20260928 23:50:05.573858 13325 heartbeater.cc:347] Connected to a master server at 127.6.152.62:36193
I20260928 23:50:05.574210 13325 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:05.574717 13325 heartbeater.cc:511] Master 127.6.152.62:36193 requested a full tablet report, sending...
I20260928 23:50:05.575819 12862 ts_manager.cc:194] Registered new tserver with Master: 24026d9eea29427bb69abcf470ecf1da (127.6.152.3:33437)
I20260928 23:50:05.576391 12862 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.152.3@KRBTEST.COM'} at 127.6.152.3:36921
I20260928 23:50:05.582396 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.152.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.152.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.152.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.61.keytab.
Entry for principal oryx/127.6.152.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.61.keytab.
I20260928 23:50:05.608240 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.61:46027
--webserver_interface=127.6.152.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.61.keytab
--principal=oryx/127.6.152.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:38547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:36193,127.6.152.61:46027
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:50:05.716549 13330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:05.716883 13330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:05.716989 13330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:05.721007 13330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:50:05.721091 13330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:05.721127 13330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:05.721163 13330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:50:05.721202 13330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:50:05.726517 13330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:38547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:36193,127.6.152.61:46027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.61.keytab
--principal=oryx/127.6.152.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.61:46027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.152.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
W20260928 23:50:05.726763 13330 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.6.152.62:36193,127.6.152.61:46027'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 23:50:05.727622 13330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:05.728605 13330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:05.737053 13338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:05.737077 13336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:05.737074 13335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:05.737279 13330 server_base.cc:1034] running on GCE node
I20260928 23:50:05.737877 13330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:05.739039 13330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:05.740232 13330 hybrid_clock.cc:648] HybridClock initialized: now 1790639405740215 us; error 31 us; skew 500 ppm
Sep 28 23:50:05 dist-test-slave-p5bw krb5kdc[12817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639405, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:05.742906 13330 init.cc:377] Logged in from keytab as oryx/127.6.152.61@KRBTEST.COM (short username oryx)
I20260928 23:50:05.743750 13330 webserver.cc:460] Webserver started at http://127.6.152.61:44595/ using document root <none> and password file <none>
I20260928 23:50:05.744284 13330 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:05.744374 13330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:05.744608 13330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:05.746672 13330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/data/instance:
uuid: "8c611b33841b41618e43a93d54e67fd7"
format_stamp: "Formatted at 2026-09-28 23:50:05 on dist-test-slave-p5bw"
server_key: "dfc3894e3d2b8c308a9f9130174e4abd"
server_key_iv: "6a8aef92af7274d9eb9c209ee08b314b"
server_key_version: "encryptionkey@0"
I20260928 23:50:05.747220 13330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/wal/instance:
uuid: "8c611b33841b41618e43a93d54e67fd7"
format_stamp: "Formatted at 2026-09-28 23:50:05 on dist-test-slave-p5bw"
server_key: "dfc3894e3d2b8c308a9f9130174e4abd"
server_key_iv: "6a8aef92af7274d9eb9c209ee08b314b"
server_key_version: "encryptionkey@0"
I20260928 23:50:05.751009 13330 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:50:05.753221 13345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:05.754359 13330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:50:05.754498 13330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/wal
uuid: "8c611b33841b41618e43a93d54e67fd7"
format_stamp: "Formatted at 2026-09-28 23:50:05 on dist-test-slave-p5bw"
server_key: "dfc3894e3d2b8c308a9f9130174e4abd"
server_key_iv: "6a8aef92af7274d9eb9c209ee08b314b"
server_key_version: "encryptionkey@0"
I20260928 23:50:05.754606 13330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:05.766423 13330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:05.769146 13330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:05.769333 13330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:05.776036 13330 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.61:46027
I20260928 23:50:05.776300 13397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.61:46027 every 8 connection(s)
I20260928 23:50:05.777036 13330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/data/info.pb
I20260928 23:50:05.779831 13398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:05.783788 13398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:50:05.784433 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 13330
Sep 28 23:50:05 dist-test-slave-p5bw krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639405, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.61@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM
Sep 28 23:50:05 dist-test-slave-p5bw krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.152.61@KRBTEST.COM
I20260928 23:50:05.798341 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:05.785796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53080 (local address 127.6.152.62:36193)
0928 23:50:05.785921 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:05.785926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:05.786645 (+ 719us) server_negotiation.cc:407] Connection header received
0928 23:50:05.787543 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:05.787546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:05.787595 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:05.787667 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:05.789056 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.789814 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.791205 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.791521 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.791627 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:05.794211 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:05.794217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:05.794221 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:05.794256 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:05.796021 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:05.796694 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:05.796697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:05.796699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:05.796787 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:05.797195 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:05.797198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:05.797199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:05.797512 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:05.797611 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:05.798067 (+ 456us) server_negotiation.cc:299] Negotiation successful
0928 23:50:05.798186 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:50:05.799207 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:05.785921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.0.0.1:53080)
0928 23:50:05.786491 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:05.786535 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:05.787253 (+ 718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:05.788253 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:05.788259 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:05.788700 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:05.788916 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:05.788927 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.789932 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.789934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:05.791086 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:05.791098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.791654 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.791657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:05.791786 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:05.792427 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:05.792447 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:05.794069 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:05.796133 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:05.796142 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:05.796148 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:05.796591 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:05.796894 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:05.796897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:05.796898 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:05.797105 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:05.797619 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:05.797624 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:05.797829 (+ 205us) client_negotiation.cc:769] Sending connection context
0928 23:50:05.798004 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 23:50:05.798597 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"spinlock_wait_cycles":61184,"thread_start_us":157,"threads_started":1}
I20260928 23:50:05.799207 13401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:05.785921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52592 (local address 127.6.152.61:46027)
0928 23:50:05.786393 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:05.786410 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:05.786449 (+ 39us) server_negotiation.cc:407] Connection header received
0928 23:50:05.787253 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:05.787275 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:05.787556 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:05.787741 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:05.788612 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.790027 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.791112 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.791646 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.791783 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:05.794732 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:05.794737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:05.794763 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:05.794789 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:05.796705 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:05.797195 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:05.797200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:05.797201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:05.797283 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:05.797533 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:05.797537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:05.797542 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:05.797820 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:05.798095 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:05.798226 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 23:50:05.798498 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":312,"thread_start_us":144,"threads_started":1}
I20260928 23:50:05.800513 13398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:50:05 dist-test-slave-p5bw krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639405, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.61@KRBTEST.COM for oryx/127.6.152.61@KRBTEST.COM
I20260928 23:50:05.807751 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:05.800966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:46027 (local address 127.0.0.1:52608)
0928 23:50:05.801077 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:05.801090 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:05.801183 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:05.801412 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:05.801414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:05.801529 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:05.801613 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:05.801618 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.802230 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.802233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:05.802775 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:05.802782 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.803607 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.803610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:05.803682 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:05.804127 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:05.804138 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:05.805265 (+ 1127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:05.806651 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:05.806653 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:05.806654 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:05.806825 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:05.807042 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:05.807044 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:05.807045 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:05.807111 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:05.807388 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:05.807390 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:05.807453 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 23:50:05.807526 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 23:50:05.807628 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 23:50:05.807785 13401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:05.801045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52608 (local address 127.6.152.61:46027)
0928 23:50:05.801141 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:05.801145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:05.801157 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:50:05.801270 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:05.801272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:05.801308 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:05.801373 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:05.801699 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.802153 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.802875 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.803515 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.803591 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:05.805382 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:05.805385 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:05.805387 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:05.805405 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:05.806544 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:05.806901 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:05.806903 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:05.806904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:05.806969 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:05.807180 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:05.807182 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:05.807183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:05.807316 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:05.807384 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:05.807576 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 23:50:05.807675 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":36}
I20260928 23:50:05.810343 13398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7: Bootstrap starting.
I20260928 23:50:05.812397 13398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:05.813237 13398 log.cc:824] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7: Log is configured to *not* fsync() on all Append() calls
I20260928 23:50:05.814880 13398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7: No bootstrap required, opened a new log
I20260928 23:50:05.817705 13398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } }
I20260928 23:50:05.817878 13398 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:05.817960 13398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c611b33841b41618e43a93d54e67fd7, State: Initialized, Role: FOLLOWER
I20260928 23:50:05.818349 13398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } }
I20260928 23:50:05.818771 13404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } } }
I20260928 23:50:05.818894 13404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:50:05.819401 13398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 23:50:05.823592 13415 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:50:05.823675 13415 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:50:05.825464 13330 master_runner.cc:428] Detected that this master 8c611b33841b41618e43a93d54e67fd7 is joining an existing cluster
I20260928 23:50:05.825508 13330 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.152.61:46027
I20260928 23:50:05.837004 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:05.827510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53088 (local address 127.6.152.62:36193)
0928 23:50:05.827636 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:05.827641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:05.828229 (+ 588us) server_negotiation.cc:407] Connection header received
0928 23:50:05.828415 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:05.828418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:05.828461 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:05.828511 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:05.829367 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.830126 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.831311 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.831633 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.831700 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:05.833336 (+ 1636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:05.833340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:05.833343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:05.833365 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:05.835187 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:05.835680 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:05.835686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:05.835688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:05.835761 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:05.836078 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:05.836081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:05.836083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:05.836304 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:05.836371 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:05.836702 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 23:50:05.836846 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 23:50:05.837036 13401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:05.827873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52616 (local address 127.6.152.61:46027)
0928 23:50:05.827979 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:05.827984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:05.828289 (+ 305us) server_negotiation.cc:407] Connection header received
0928 23:50:05.828414 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:05.828417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:05.828452 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:05.828499 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:05.829355 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.830119 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.831437 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.831740 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.831807 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:05.833325 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:05.833329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:05.833331 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:05.833352 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:05.835490 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:05.835928 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:05.835930 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:05.835932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:05.835991 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:05.836245 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:05.836247 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:05.836249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:05.836434 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:05.836491 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:05.836792 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 23:50:05.836913 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 23:50:05.837531 13421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:05.827863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:46027 (local address 127.0.0.1:52616)
0928 23:50:05.828226 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:05.828238 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:05.828327 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:05.828634 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:05.828637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:05.828966 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:05.829238 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:05.829245 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.830253 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.830255 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:05.831346 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:05.831353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.831921 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.831925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:05.832053 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:05.832696 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:05.832711 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:05.833196 (+ 485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:05.835613 (+ 2417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:05.835619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:05.835621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:05.835841 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:05.836085 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:05.836089 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:05.836090 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:05.836159 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:05.836545 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:05.836547 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:05.836643 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 23:50:05.837185 (+ 542us) client_negotiation.cc:240] Negotiation successful
0928 23:50:05.837339 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":142,"spinlock_wait_cycles":5504,"thread_start_us":126,"threads_started":1}
I20260928 23:50:05.837541 13422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:05.827428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.0.0.1:53088)
0928 23:50:05.828148 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:05.828164 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:05.828303 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:05.828644 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:05.828647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:05.828967 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:05.829238 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:05.829245 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.830247 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.830250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:05.831183 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:05.831193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.831855 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.831858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:05.831978 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:05.832574 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:05.832592 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:05.833196 (+ 604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:05.835313 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:05.835318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:05.835321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:05.835574 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:05.835874 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:05.835880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:05.835883 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:05.835963 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:05.836435 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:05.836438 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:05.836521 (+ 83us) client_negotiation.cc:769] Sending connection context
0928 23:50:05.837185 (+ 664us) client_negotiation.cc:240] Negotiation successful
0928 23:50:05.837388 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":188,"spinlock_wait_cycles":5120,"thread_start_us":108,"threads_started":1}
Sep 28 23:50:05 dist-test-slave-p5bw krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639404, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.62@KRBTEST.COM for oryx/127.6.152.61@KRBTEST.COM
I20260928 23:50:05.850492 13423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:05.841565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:46027 (local address 127.0.0.1:52626)
0928 23:50:05.841845 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:05.841888 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:05.842011 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:05.842309 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:05.842314 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:05.842613 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:05.842703 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:05.842707 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.843657 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.843659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:05.844256 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:05.844263 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.844791 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.844794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:05.844938 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:05.845440 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:05.845452 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:05.846914 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:05.848724 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:05.848732 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:05.848736 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:05.849034 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:05.849383 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:05.849388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:05.849390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:05.849471 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:05.849848 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:05.849857 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:05.849964 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 23:50:05.850121 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 23:50:05.850301 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":64,"threads_started":1}
I20260928 23:50:05.850489 13401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:05.841565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52626 (local address 127.6.152.61:46027)
0928 23:50:05.841664 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:05.841670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:05.841972 (+ 302us) server_negotiation.cc:407] Connection header received
0928 23:50:05.842144 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:05.842147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:05.842197 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:05.842288 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:05.842804 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.843556 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.844359 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.844691 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.844787 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:05.847037 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:05.847041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:05.847044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:05.847067 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:05.848612 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:05.849164 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:05.849169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:05.849171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:05.849251 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:05.849570 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:05.849573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:05.849575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:05.849735 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:05.849837 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:05.850191 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 23:50:05.850297 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 23:50:05.851866 12862 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.152.61:46027: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8c611b33841b41618e43a93d54e67fd7"
member_type: NON_VOTER
last_known_addr {
host: "127.6.152.61"
port: 46027
}
attrs {
promote: true
}
}
dest_uuid: "8b20d904b67749e9a3276bfdd467c7be"
cas_config_opid_index: -1
I20260928 23:50:05.852542 12862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: NON_VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: true } }
I20260928 23:50:05.855199 13372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:05.856597 13372 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [term 1 FOLLOWER]: Refusing update from remote peer 8b20d904b67749e9a3276bfdd467c7be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 23:50:05.857225 13426 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: NON_VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:50:05.854596 13424 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8c611b33841b41618e43a93d54e67fd7 (127.6.152.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: NON_VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: true } } }
I20260928 23:50:05.858789 13424 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.152.61:46027
I20260928 23:50:05.859200 13330 master.cc:537] Master@127.6.152.61:46027 shutting down...
I20260928 23:50:05.859515 13426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8b20d904b67749e9a3276bfdd467c7be" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: NON_VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: true } } }
I20260928 23:50:05.859588 13426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:05.863201 13372 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8c611b33841b41618e43a93d54e67fd7 (127.6.152.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: NON_VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: true } } }
I20260928 23:50:05.864411 13429 mvcc.cc:205] Tried to move back new op lower bound from 7334459002405638144 to 7334459002258513920. Current Snapshot: MvccSnapshot[applied={T|T < 7334459002294771712}]
I20260928 23:50:05.865262 13404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b20d904b67749e9a3276bfdd467c7be. Latest consensus state: current_term: 1 leader_uuid: "8b20d904b67749e9a3276bfdd467c7be" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: NON_VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: true } } }
WARNING: no policy specified for oryx/127.6.152.60@KRBTEST.COM; defaulting to no policy
I20260928 23:50:05.865353 13404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:50:05.865444 13404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b20d904b67749e9a3276bfdd467c7be" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: NON_VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: true } } }
I20260928 23:50:05.865501 13404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:50:05.865556 13404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b20d904b67749e9a3276bfdd467c7be" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: NON_VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: true } } }
I20260928 23:50:05.865613 13404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [sys.catalog]: This master's current role is: LEARNER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.152.60@KRBTEST.COM" created.
I20260928 23:50:05.870968 13330 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [term 1 LEARNER]: Raft consensus shutting down.
I20260928 23:50:05.871277 13330 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [term 1 LEARNER]: Raft consensus is shut down!
I20260928 23:50:05.871357 13330 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.152.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.60.keytab.
Entry for principal oryx/127.6.152.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.60.keytab.
I20260928 23:50:05.887892 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.60:34469
--webserver_interface=127.6.152.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.60.keytab
--principal=oryx/127.6.152.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:38547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:36193,127.6.152.61:46027,127.6.152.60:34469
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:50:05.888190 13330 master.cc:552] Master@127.6.152.61:46027 shutdown complete.
I20260928 23:50:05.888284 13330 master_runner.cc:305] Clearing existing system tablet
I20260928 23:50:05.888573 13330 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:05.888669 13330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:05.890085 13330 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:50:05.893301 13439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:05.893622 13330 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:05.893759 13330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260928 23:50:05.893800 13330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/wal
uuid: "8c611b33841b41618e43a93d54e67fd7"
format_stamp: "Formatted at 2026-09-28 23:50:05 on dist-test-slave-p5bw"
server_key: "dfc3894e3d2b8c308a9f9130174e4abd"
server_key_iv: "6a8aef92af7274d9eb9c209ee08b314b"
server_key_version: "encryptionkey@0"
I20260928 23:50:05.896355 13330 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:50:05.899151 13330 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 23:50:05.899261 13330 log.cc:1196] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 23:50:05.899617 13330 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7: Deleting consensus metadata
I20260928 23:50:05.899767 13330 master_runner.cc:315] Copying system tablet from 127.6.152.62:36193
I20260928 23:50:05.900856 13330 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7: tablet copy: Beginning tablet copy session from remote peer at address 127.6.152.62:36193
I20260928 23:50:05.911759 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:05.901443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53090 (local address 127.6.152.62:36193)
0928 23:50:05.901582 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:05.901588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:05.905099 (+ 3511us) server_negotiation.cc:407] Connection header received
0928 23:50:05.905267 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:05.905270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:05.905324 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:05.905408 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:05.905839 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.906378 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.907194 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.907423 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.907530 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:05.908634 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:05.908638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:05.908640 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:05.908664 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:05.910175 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:05.910595 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:05.910600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:05.910601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:05.910680 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:05.910951 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:05.910956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:05.910958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:05.911135 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:05.911238 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:05.911489 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 23:50:05.911600 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:50:05.911777 13445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:05.901392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.0.0.1:53090)
0928 23:50:05.904989 (+ 3597us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:05.905011 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:05.905147 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:05.905426 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:05.905429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:05.905632 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:05.905723 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:05.905729 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.906488 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.906492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:05.907077 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:05.907084 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:05.907535 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:05.907539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:05.907635 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:05.908166 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:05.908177 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:05.908486 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:05.910295 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:05.910298 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:05.910300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:05.910493 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:05.910777 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:05.910780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:05.910782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:05.910864 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:05.911228 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:05.911231 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:05.911333 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 23:50:05.911437 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 23:50:05.911587 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3530,"thread_start_us":82,"threads_started":1}
I20260928 23:50:05.912389 12882 tablet_copy_service.cc:140] P 8b20d904b67749e9a3276bfdd467c7be: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8c611b33841b41618e43a93d54e67fd7 ({username='oryx', principal='oryx/127.6.152.61@KRBTEST.COM'} at 127.0.0.1:53090)
I20260928 23:50:05.912591 12882 tablet_copy_service.cc:161] P 8b20d904b67749e9a3276bfdd467c7be: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8c611b33841b41618e43a93d54e67fd7 at {username='oryx', principal='oryx/127.6.152.61@KRBTEST.COM'} at 127.0.0.1:53090: session id = 8c611b33841b41618e43a93d54e67fd7-00000000000000000000000000000000
I20260928 23:50:05.914466 12882 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 23:50:05.915952 13330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 23:50:05.917795 13330 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7: tablet copy: Starting download of 0 data blocks...
I20260928 23:50:05.918000 13330 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7: tablet copy: Starting download of 1 WAL segments...
I20260928 23:50:05.919638 13330 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 23:50:05.921468 12882 tablet_copy_service.cc:342] P 8b20d904b67749e9a3276bfdd467c7be: Request end of tablet copy session 8c611b33841b41618e43a93d54e67fd7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.152.61@KRBTEST.COM'} at 127.0.0.1:53090
I20260928 23:50:05.921597 12882 tablet_copy_service.cc:433] P 8b20d904b67749e9a3276bfdd467c7be: ending tablet copy session 8c611b33841b41618e43a93d54e67fd7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8c611b33841b41618e43a93d54e67fd7
I20260928 23:50:05.927955 13330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:05.929626 13449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:05.929622 13446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:05.932991 13447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:05.933857 13330 server_base.cc:1034] running on GCE node
I20260928 23:50:05.934034 13330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:05.934331 13330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:05.935487 13330 hybrid_clock.cc:648] HybridClock initialized: now 1790639405935505 us; error 52 us; skew 500 ppm
Sep 28 23:50:05 dist-test-slave-p5bw krb5kdc[12817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639405, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:05.937072 13330 init.cc:377] Logged in from keytab as oryx/127.6.152.61@KRBTEST.COM (short username oryx)
I20260928 23:50:05.937534 13330 webserver.cc:460] Webserver started at http://127.6.152.61:42135/ using document root <none> and password file <none>
I20260928 23:50:05.937719 13330 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:05.937774 13330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:05.938879 13330 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:50:05.939772 13456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:05.940002 13330 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:50:05.940074 13330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/wal
uuid: "8c611b33841b41618e43a93d54e67fd7"
format_stamp: "Formatted at 2026-09-28 23:50:05 on dist-test-slave-p5bw"
server_key: "dfc3894e3d2b8c308a9f9130174e4abd"
server_key_iv: "6a8aef92af7274d9eb9c209ee08b314b"
server_key_version: "encryptionkey@0"
I20260928 23:50:05.940157 13330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:05.947576 13330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:05.949771 13330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:05.955824 13330 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.61:46027
I20260928 23:50:05.956508 13330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-1/data/info.pb
I20260928 23:50:05.957937 13510 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:50:05.958748 13510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7: Bootstrap starting.
I20260928 23:50:05.960964 13509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.61:46027 every 8 connection(s)
I20260928 23:50:05.964969 13510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:50:05.965260 13510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7: Bootstrap complete.
I20260928 23:50:05.965741 13510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: NON_VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: true } }
I20260928 23:50:05.965868 13510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8c611b33841b41618e43a93d54e67fd7, State: Initialized, Role: LEARNER
I20260928 23:50:05.965992 13510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: NON_VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: true } }
I20260928 23:50:05.966295 13513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: NON_VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: true } } }
I20260928 23:50:05.966451 13513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:50:05.966459 13510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:50:05.969334 13524 catalog_manager.cc:1259] Loaded cluster ID: cef39238cd1d47c9b5fe97b10e1b2ace
I20260928 23:50:05.969449 13524 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7: loading cluster ID for follower catalog manager: success
I20260928 23:50:05.971046 13524 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7: acquiring CA information for follower catalog manager: success
I20260928 23:50:05.972105 13524 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 23:50:06.041846 13437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:06.042100 13437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:06.042182 13437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:06.046154 13437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:50:06.046211 13437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:06.046231 13437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:06.046247 13437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:50:06.046311 13437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:50:06.051805 13437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:38547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:36193,127.6.152.61:46027,127.6.152.60:34469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.60.keytab
--principal=oryx/127.6.152.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.60:34469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.152.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:06.052882 13437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:06.053887 13437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:06.062139 13532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:06.062148 13529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:06.062148 13530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:06.062536 13437 server_base.cc:1034] running on GCE node
I20260928 23:50:06.063135 13437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:06.064325 13437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:06.065523 13437 hybrid_clock.cc:648] HybridClock initialized: now 1790639406065504 us; error 30 us; skew 500 ppm
Sep 28 23:50:06 dist-test-slave-p5bw krb5kdc[12817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639406, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:06.068775 13437 init.cc:377] Logged in from keytab as oryx/127.6.152.60@KRBTEST.COM (short username oryx)
I20260928 23:50:06.069685 13437 webserver.cc:460] Webserver started at http://127.6.152.60:43255/ using document root <none> and password file <none>
I20260928 23:50:06.070245 13437 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:06.070331 13437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:06.070547 13437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:06.072647 13437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/data/instance:
uuid: "ec71e3446d104e96b5462e085b9e757b"
format_stamp: "Formatted at 2026-09-28 23:50:06 on dist-test-slave-p5bw"
server_key: "a8da7d7dbfe20726fe14a998a7b31070"
server_key_iv: "27995e9555cbaf3d9b81f542ebd8b3ea"
server_key_version: "encryptionkey@0"
I20260928 23:50:06.073336 13437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/wal/instance:
uuid: "ec71e3446d104e96b5462e085b9e757b"
format_stamp: "Formatted at 2026-09-28 23:50:06 on dist-test-slave-p5bw"
server_key: "a8da7d7dbfe20726fe14a998a7b31070"
server_key_iv: "27995e9555cbaf3d9b81f542ebd8b3ea"
server_key_version: "encryptionkey@0"
I20260928 23:50:06.077050 13437 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:50:06.079411 13539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:06.080677 13437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:50:06.080888 13437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/wal
uuid: "ec71e3446d104e96b5462e085b9e757b"
format_stamp: "Formatted at 2026-09-28 23:50:06 on dist-test-slave-p5bw"
server_key: "a8da7d7dbfe20726fe14a998a7b31070"
server_key_iv: "27995e9555cbaf3d9b81f542ebd8b3ea"
server_key_version: "encryptionkey@0"
I20260928 23:50:06.080992 13437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:06.102421 13437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:06.105672 13437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:06.105846 13437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:06.112650 13437 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.60:34469
I20260928 23:50:06.112941 13591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.60:34469 every 8 connection(s)
I20260928 23:50:06.113641 13437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/data/info.pb
I20260928 23:50:06.115654 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 13437
I20260928 23:50:06.116434 13592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:06.121261 13592 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:50:06 dist-test-slave-p5bw krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.152.60@KRBTEST.COM
I20260928 23:50:06.125905 13054 heartbeater.cc:503] Master 127.6.152.62:36193 was elected leader, sending a full tablet report...
Sep 28 23:50:06 dist-test-slave-p5bw krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639406, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.60@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM
I20260928 23:50:06.131124 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.117330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39140 (local address 127.6.152.60:34469)
0928 23:50:06.117746 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.117765 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.117805 (+ 40us) server_negotiation.cc:407] Connection header received
0928 23:50:06.118572 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.118596 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.118999 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.119359 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:06.120024 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.121485 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.122392 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.122966 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.123133 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.126089 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:06.126095 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:06.126107 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:06.126135 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:06.128391 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.128832 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.128839 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.128845 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.129017 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.129298 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.129301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.129303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.129682 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:06.129897 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.130141 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.130406 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":120,"threads_started":1}
I20260928 23:50:06.132522 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.123169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53092 (local address 127.6.152.62:36193)
0928 23:50:06.123300 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.123304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.123627 (+ 323us) server_negotiation.cc:407] Connection header received
0928 23:50:06.123838 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.123840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.123890 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.123957 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:06.124876 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.125376 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.126470 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.126745 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.126847 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.129399 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:06.129407 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:06.129409 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:06.129449 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:06.130845 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.131327 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.131330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.131332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.131423 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.131674 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.131678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.131680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.131842 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:06.131949 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.132256 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.132358 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 23:50:06.132539 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.123072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.0.0.1:53092)
0928 23:50:06.123503 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:06.123541 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:06.123669 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:06.124045 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:06.124051 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:06.124534 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:06.124708 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.124716 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.125510 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.125514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.126359 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.126371 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.126871 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.126875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.127003 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.127743 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:06.127761 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:06.129258 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:06.130968 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.130975 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.130981 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.131194 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.131516 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.131518 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.131519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.131584 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.131952 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:06.131960 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:06.132060 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 23:50:06.132191 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 23:50:06.132359 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":149,"threads_started":1}
I20260928 23:50:06.133587 13592 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:50:06 dist-test-slave-p5bw krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639406, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.60@KRBTEST.COM for oryx/127.6.152.61@KRBTEST.COM
I20260928 23:50:06.141542 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.134025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:46027 (local address 127.0.0.1:52642)
0928 23:50:06.134137 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:06.134150 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:06.134219 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:06.134618 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:06.134621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:06.134779 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:06.134855 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.134860 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.135800 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.135802 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.136315 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.136322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.136882 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.136885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.136964 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.137317 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:06.137327 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:06.138412 (+ 1085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:06.140145 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.140148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.140150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.140335 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.140615 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.140618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.140619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.140697 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.141083 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:06.141086 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:06.141174 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 23:50:06.141275 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 23:50:06.141391 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 23:50:06.141594 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.134162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52642 (local address 127.6.152.61:46027)
0928 23:50:06.134406 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.134423 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.134434 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:50:06.134468 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.134470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.134507 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.134606 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:06.134972 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.135688 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.136414 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.136720 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.136865 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.138550 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:06.138553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:06.138555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:06.138577 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:06.140020 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.140437 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.140441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.140442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.140517 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.140782 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.140787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.140789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.140982 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:06.141082 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.141323 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.141444 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
I20260928 23:50:06.142330 13592 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.60" port: 34469 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:50:06 dist-test-slave-p5bw krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639406, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.60@KRBTEST.COM for oryx/127.6.152.60@KRBTEST.COM
I20260928 23:50:06.149631 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.142835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39148 (local address 127.6.152.60:34469)
0928 23:50:06.142964 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.142967 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.142978 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:50:06.143003 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.143005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.143035 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.143094 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:06.143384 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.144180 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.144933 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.145135 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.145208 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.146803 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:06.146806 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:06.146808 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:06.146826 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:06.148117 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.148513 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.148519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.148521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.148601 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.148886 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.148893 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.148894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.149046 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:06.149150 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.149380 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.149511 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":42}
I20260928 23:50:06.149631 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.142748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:34469 (local address 127.0.0.1:39148)
0928 23:50:06.142824 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:06.142834 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:06.142902 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:06.143119 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:06.143121 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:06.143237 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:06.143299 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.143303 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.144285 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.144287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.144771 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.144778 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.145220 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.145222 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.145277 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.145608 (+ 331us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:06.145618 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:06.146672 (+ 1054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:06.148238 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.148242 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.148244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.148422 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.148691 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.148693 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.148695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.148770 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.149148 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:06.149151 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:06.149226 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 23:50:06.149330 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 23:50:06.149457 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 23:50:06.152343 13592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b: Bootstrap starting.
I20260928 23:50:06.154534 13592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:06.155174 13592 log.cc:824] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b: Log is configured to *not* fsync() on all Append() calls
I20260928 23:50:06.156970 13592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b: No bootstrap required, opened a new log
I20260928 23:50:06.159832 13592 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 34469 } }
I20260928 23:50:06.160019 13592 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:06.160053 13592 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec71e3446d104e96b5462e085b9e757b, State: Initialized, Role: FOLLOWER
I20260928 23:50:06.160465 13592 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 34469 } }
I20260928 23:50:06.160894 13598 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 34469 } } }
I20260928 23:50:06.161027 13598 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:50:06.161602 13592 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [sys.catalog]: configured and running, proceeding with master startup.
W20260928 23:50:06.165987 13609 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:50:06.166081 13609 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:50:06.169764 13437 master_runner.cc:428] Detected that this master ec71e3446d104e96b5462e085b9e757b is joining an existing cluster
I20260928 23:50:06.169827 13437 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.152.60:34469
I20260928 23:50:06.181056 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.172053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.0.0.1:53102)
0928 23:50:06.172746 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:06.172760 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:06.172912 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:06.173206 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:06.173209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:06.173528 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:06.173664 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.173671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.174813 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.174816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.175758 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.175769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.176483 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.176487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.176599 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.177444 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:06.177457 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:06.177878 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:06.179517 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.179521 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.179523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.179787 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.180075 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.180078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.180080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.180170 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.180511 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:06.180514 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:06.180599 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 23:50:06.180710 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 23:50:06.180860 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":2,"thread_start_us":97,"threads_started":1}
I20260928 23:50:06.181056 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.172103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53102 (local address 127.6.152.62:36193)
0928 23:50:06.172245 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.172250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.172874 (+ 624us) server_negotiation.cc:407] Connection header received
0928 23:50:06.173023 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.173028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.173088 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.173170 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:06.173806 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.174650 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.175892 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.176245 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.176321 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.178042 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:06.178045 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:06.178047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:06.178069 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:06.179408 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.179884 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.179886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.179888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.179990 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.180255 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.180257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.180258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.180423 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:06.180498 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.180740 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.180937 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:50:06.184299 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 12924
Sep 28 23:50:06 dist-test-slave-p5bw krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639404, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.62@KRBTEST.COM for oryx/127.6.152.60@KRBTEST.COM
I20260928 23:50:06.193429 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:38171
--local_ip_for_outbound_sockets=127.6.152.1
--tserver_master_addrs=127.6.152.62:36193
--webserver_port=36617
--webserver_interface=127.6.152.1
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.1.keytab
--principal=oryx/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:38547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.152.62:36193,127.6.152.61:46027
--tserver_master_addrs=127.6.152.62:36193,127.6.152.61:46027,127.6.152.60:34469 with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:50:06.196748 13423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.185540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:34469 (local address 127.0.0.1:39162)
0928 23:50:06.185791 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:06.185807 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:06.185885 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:06.186254 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:06.186259 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:06.186496 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:06.186622 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.186632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.187718 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.187724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.188739 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.188752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.189375 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.189382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.189515 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.189998 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:06.190017 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:06.191670 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:06.194938 (+ 3268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.194942 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.194944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.195217 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.195566 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.195571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.195574 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.195669 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.196206 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:06.196209 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:06.196313 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 23:50:06.196401 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 23:50:06.196557 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260928 23:50:06.197440 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.185760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39162 (local address 127.6.152.60:34469)
0928 23:50:06.185996 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.186003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.186016 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:50:06.186075 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.186078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.186126 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.186219 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:06.186740 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.187527 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.188900 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.189249 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.189347 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.191823 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:06.191827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:06.191830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:06.191853 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:06.194771 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.195353 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.195358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.195362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.195455 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.195795 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.195801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.195804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.196063 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:06.197063 (+ 1000us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.197135 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.197272 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":82,"threads_started":1}
I20260928 23:50:06.200133 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.172491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39150 (local address 127.6.152.60:34469)
0928 23:50:06.172578 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.172583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.173280 (+ 697us) server_negotiation.cc:407] Connection header received
0928 23:50:06.173391 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.173394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.173430 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.173516 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:06.174126 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.174872 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.176019 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.176329 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.176392 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.196001 (+ 19609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:06.196008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:06.196012 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:06.196037 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:06.198660 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.199084 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.199088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.199090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.199150 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.199381 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.199384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.199386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.199550 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:06.199614 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.199882 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.199995 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 23:50:06.200745 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.172478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:34469 (local address 127.0.0.1:39150)
0928 23:50:06.173215 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:06.173227 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:06.173323 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:06.173609 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:06.173615 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:06.173883 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:06.174008 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.174018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.174978 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.174981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.175904 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.175915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.176516 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.176519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.176694 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.194952 (+ 18258us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:06.194966 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:06.195627 (+ 661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:06.198771 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.198774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.198776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.198993 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.199236 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.199238 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.199240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.199303 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.199670 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:06.199672 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:06.199739 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 23:50:06.200461 (+ 722us) client_negotiation.cc:240] Negotiation successful
0928 23:50:06.200584 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":69,"thread_start_us":76,"threads_started":1}
I20260928 23:50:06.201579 12862 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.152.60:34469: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ec71e3446d104e96b5462e085b9e757b"
member_type: NON_VOTER
last_known_addr {
host: "127.6.152.60"
port: 34469
}
attrs {
promote: true
}
}
dest_uuid: "8b20d904b67749e9a3276bfdd467c7be"
cas_config_opid_index: 5
I20260928 23:50:06.202219 12862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: NON_VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: true } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: NON_VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: true } }
I20260928 23:50:06.203444 13427 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ec71e3446d104e96b5462e085b9e757b (127.6.152.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: NON_VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: true } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: NON_VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: true } } }
I20260928 23:50:06.204106 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.172061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:46027 (local address 127.0.0.1:52650)
0928 23:50:06.172985 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:06.173020 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:06.173414 (+ 394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:06.173764 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:06.173767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:06.174032 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:06.174143 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.174149 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.175338 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.175340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.176680 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.176687 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.177237 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.177241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.177330 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.193471 (+ 16141us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:06.193486 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:06.194472 (+ 986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:06.197159 (+ 2687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.197162 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.197164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.197401 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.202689 (+ 5288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.202694 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.202695 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.202807 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.203353 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:06.203356 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:06.203448 (+ 92us) client_negotiation.cc:769] Sending connection context
0928 23:50:06.203581 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 23:50:06.203743 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"mutex_wait_us":41,"thread_start_us":112,"threads_started":1}
I20260928 23:50:06.204843 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.172437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52650 (local address 127.6.152.61:46027)
0928 23:50:06.172562 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.172566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.173088 (+ 522us) server_negotiation.cc:407] Connection header received
0928 23:50:06.173519 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.173522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.173570 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.173623 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:06.174254 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.175027 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.176826 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.177118 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.177232 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.194993 (+ 17761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:06.194998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:06.195001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:06.195032 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:06.196686 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.202443 (+ 5757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.202486 (+ 43us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.202488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.202585 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.202955 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.202960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.202965 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.203211 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:06.203307 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.204419 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.204573 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:50:06.204903 13427 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.152.60:34469
I20260928 23:50:06.205084 13427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8b20d904b67749e9a3276bfdd467c7be" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: NON_VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: true } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: NON_VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: true } } }
I20260928 23:50:06.205173 13427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:06.205328 13437 master.cc:537] Master@127.6.152.60:34469 shutting down...
W20260928 23:50:06.206996 12849 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 23:50:06.208005 12849 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be -> Peer ec71e3446d104e96b5462e085b9e757b (127.6.152.60:34469): Couldn't send request to peer ec71e3446d104e96b5462e085b9e757b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 23:50:06.210714 13437 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [term 0 FOLLOWER]: Raft consensus shutting down.
I20260928 23:50:06.211014 13437 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [term 0 FOLLOWER]: Raft consensus is shut down!
I20260928 23:50:06.211119 13437 tablet_replica.cc:330] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b: stopping tablet replica
I20260928 23:50:06.213788 13423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.205503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:46027 (local address 127.0.0.1:52666)
0928 23:50:06.208507 (+ 3004us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:06.208522 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:06.208615 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:06.208962 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:06.208965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:06.208985 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:06.209079 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.209088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.210165 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.210172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.211831 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.211845 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.213223 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.213229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.213359 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.213423 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 23:50:06.213512 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 23:50:06.213610 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2915}
I20260928 23:50:06.214248 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.207574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52666 (local address 127.6.152.61:46027)
0928 23:50:06.207724 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.207731 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.208603 (+ 872us) server_negotiation.cc:407] Connection header received
0928 23:50:06.208701 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.208707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.208783 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.208940 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:06.209219 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.210037 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.212014 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.212873 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.212980 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.213151 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.213986 (+ 835us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.214089 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 23:50:06.215093 13484 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [term 1 LEARNER]: Refusing update from remote peer 8b20d904b67749e9a3276bfdd467c7be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 23:50:06.215579 13426 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: NON_VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 23:50:06.216648 13484 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ec71e3446d104e96b5462e085b9e757b (127.6.152.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: NON_VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: true } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: NON_VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: true } } }
I20260928 23:50:06.218230 13513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b20d904b67749e9a3276bfdd467c7be. Latest consensus state: current_term: 1 leader_uuid: "8b20d904b67749e9a3276bfdd467c7be" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: NON_VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: true } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: NON_VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: true } } }
I20260928 23:50:06.218330 13513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:50:06.218276 13623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b20d904b67749e9a3276bfdd467c7be" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: NON_VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: true } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: NON_VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: true } } }
I20260928 23:50:06.218393 13623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:50:06.226016 13424 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be: attempting to promote NON_VOTER 8c611b33841b41618e43a93d54e67fd7 to VOTER
I20260928 23:50:06.226496 13424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: NON_VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: true } }
I20260928 23:50:06.227662 13484 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [term 1 LEARNER]: Refusing update from remote peer 8b20d904b67749e9a3276bfdd467c7be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 23:50:06.227837 13423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.227472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:34469 (local address 127.0.0.1:39164)
0928 23:50:06.227628 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:06.227698 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:34469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 23:50:06.228036 13426 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 23:50:06.228669 13423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.228444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:34469 (local address 127.0.0.1:39176)
0928 23:50:06.228568 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:06.228612 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:34469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260928 23:50:06.228843 12849 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be -> Peer ec71e3446d104e96b5462e085b9e757b (127.6.152.60:34469): Couldn't send request to peer ec71e3446d104e96b5462e085b9e757b. Status: Network error: Client connection negotiation failed: client connection to 127.6.152.60:34469: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 23:50:06.230037 13437 master.cc:552] Master@127.6.152.60:34469 shutdown complete.
I20260928 23:50:06.230088 13437 master_runner.cc:305] Clearing existing system tablet
I20260928 23:50:06.230156 13426 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8c611b33841b41618e43a93d54e67fd7 (127.6.152.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: NON_VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: true } } }
I20260928 23:50:06.230320 13437 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:06.230363 13437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:06.231580 13426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8b20d904b67749e9a3276bfdd467c7be" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: NON_VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: true } } }
I20260928 23:50:06.231686 13426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:06.231799 13426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b20d904b67749e9a3276bfdd467c7be" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: NON_VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: true } } }
I20260928 23:50:06.231880 13426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:06.232424 13437 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:50:06.232417 13484 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8c611b33841b41618e43a93d54e67fd7 (127.6.152.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: NON_VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: true } } }
I20260928 23:50:06.233258 13625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:06.233429 13437 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:06.233536 13437 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 23:50:06.233570 13437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/wal
uuid: "ec71e3446d104e96b5462e085b9e757b"
format_stamp: "Formatted at 2026-09-28 23:50:06 on dist-test-slave-p5bw"
server_key: "a8da7d7dbfe20726fe14a998a7b31070"
server_key_iv: "27995e9555cbaf3d9b81f542ebd8b3ea"
server_key_version: "encryptionkey@0"
I20260928 23:50:06.234576 13623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b20d904b67749e9a3276bfdd467c7be" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: NON_VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: true } } }
I20260928 23:50:06.235330 13623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:50:06.235544 13437 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:50:06.238047 13437 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 23:50:06.238168 13437 log.cc:1196] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 23:50:06.238487 13437 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b: Deleting consensus metadata
I20260928 23:50:06.238653 13437 master_runner.cc:315] Copying system tablet from 127.6.152.62:36193
I20260928 23:50:06.239814 13437 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.152.62:36193
I20260928 23:50:06.249293 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.240759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53104 (local address 127.6.152.62:36193)
0928 23:50:06.240915 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.240919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.241138 (+ 219us) server_negotiation.cc:407] Connection header received
0928 23:50:06.241253 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.241256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.241309 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.241408 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:06.241913 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.242731 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.243918 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.244267 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.244371 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.245687 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:06.245691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:06.245695 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:06.245723 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:06.247446 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.247915 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.247919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.247921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.248007 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.248312 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.248316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.248318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.248564 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:06.248687 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.248990 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.249127 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 23:50:06.249311 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.240683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.0.0.1:53104)
0928 23:50:06.241041 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:06.241056 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:06.241169 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:06.241443 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:06.241446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:06.241661 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:06.241788 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.241795 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.242870 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.242873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.243788 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.243800 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.244390 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.244394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.244489 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.245101 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:06.245115 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:06.245538 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:06.247560 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.247564 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.247566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.247814 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.248120 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.248123 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.248125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.248215 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.248680 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:06.248683 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:06.248781 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 23:50:06.248930 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 23:50:06.249080 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":72,"threads_started":1}
I20260928 23:50:06.249990 12882 tablet_copy_service.cc:140] P 8b20d904b67749e9a3276bfdd467c7be: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ec71e3446d104e96b5462e085b9e757b ({username='oryx', principal='oryx/127.6.152.60@KRBTEST.COM'} at 127.0.0.1:53104)
I20260928 23:50:06.250090 12882 tablet_copy_service.cc:161] P 8b20d904b67749e9a3276bfdd467c7be: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ec71e3446d104e96b5462e085b9e757b at {username='oryx', principal='oryx/127.6.152.60@KRBTEST.COM'} at 127.0.0.1:53104: session id = ec71e3446d104e96b5462e085b9e757b-00000000000000000000000000000000
I20260928 23:50:06.251250 12882 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 23:50:06.252548 13437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 23:50:06.255043 13437 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b: tablet copy: Starting download of 0 data blocks...
I20260928 23:50:06.255239 13437 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b: tablet copy: Starting download of 1 WAL segments...
I20260928 23:50:06.256748 13437 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 23:50:06.258968 12882 tablet_copy_service.cc:342] P 8b20d904b67749e9a3276bfdd467c7be: Request end of tablet copy session ec71e3446d104e96b5462e085b9e757b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.152.60@KRBTEST.COM'} at 127.0.0.1:53104
I20260928 23:50:06.259053 12882 tablet_copy_service.cc:433] P 8b20d904b67749e9a3276bfdd467c7be: ending tablet copy session ec71e3446d104e96b5462e085b9e757b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ec71e3446d104e96b5462e085b9e757b
I20260928 23:50:06.266640 13437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:06.268555 13632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:06.269757 13633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:06.269999 13437 server_base.cc:1034] running on GCE node
W20260928 23:50:06.273016 13635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:06.273257 13437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:06.273578 13437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:06.274725 13437 hybrid_clock.cc:648] HybridClock initialized: now 1790639406274704 us; error 40 us; skew 500 ppm
Sep 28 23:50:06 dist-test-slave-p5bw krb5kdc[12817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639406, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:06.276328 13437 init.cc:377] Logged in from keytab as oryx/127.6.152.60@KRBTEST.COM (short username oryx)
I20260928 23:50:06.276793 13437 webserver.cc:460] Webserver started at http://127.6.152.60:40749/ using document root <none> and password file <none>
I20260928 23:50:06.277026 13437 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:06.277071 13437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:06.278365 13437 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:50:06.279368 13642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:06.279595 13437 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:50:06.279703 13437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/wal
uuid: "ec71e3446d104e96b5462e085b9e757b"
format_stamp: "Formatted at 2026-09-28 23:50:06 on dist-test-slave-p5bw"
server_key: "a8da7d7dbfe20726fe14a998a7b31070"
server_key_iv: "27995e9555cbaf3d9b81f542ebd8b3ea"
server_key_version: "encryptionkey@0"
I20260928 23:50:06.279846 13437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:06.293035 13437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:06.295307 13437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:06.300407 13437 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.60:34469
I20260928 23:50:06.300434 13694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.60:34469 every 8 connection(s)
I20260928 23:50:06.301147 13437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/master-2/data/info.pb
I20260928 23:50:06.303495 13695 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:50:06.304764 13695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b: Bootstrap starting.
I20260928 23:50:06.316673 13695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:50:06.317224 13695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b: Bootstrap complete.
I20260928 23:50:06.317939 13695 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: NON_VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: true } }
I20260928 23:50:06.318197 13695 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ec71e3446d104e96b5462e085b9e757b, State: Initialized, Role: LEARNER
I20260928 23:50:06.318358 13695 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: NON_VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: true } }
I20260928 23:50:06.318919 13695 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:50:06.319554 13698 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: NON_VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: true } } }
I20260928 23:50:06.319684 13698 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [sys.catalog]: This master's current role is: LEARNER
I20260928 23:50:06.322085 13709 catalog_manager.cc:1259] Loaded cluster ID: cef39238cd1d47c9b5fe97b10e1b2ace
I20260928 23:50:06.322134 13709 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b: loading cluster ID for follower catalog manager: success
I20260928 23:50:06.324353 13709 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b: acquiring CA information for follower catalog manager: success
I20260928 23:50:06.325681 13709 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:50:06.336560 13190 heartbeater.cc:503] Master 127.6.152.62:36193 was elected leader, sending a full tablet report...
W20260928 23:50:06.347947 13620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:06.348230 13620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:06.348328 13620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:06.352757 13620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:06.352861 13620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:06.352970 13620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:50:06.358129 13620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:38547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.1.keytab
--principal=oryx/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:38171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=36617
--tserver_master_addrs=127.6.152.62:36193,127.6.152.61:46027,127.6.152.60:34469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:06.359153 13620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:06.360189 13620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:06.369702 13714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:06.369702 13715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:06.369807 13717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:06.370009 13620 server_base.cc:1034] running on GCE node
I20260928 23:50:06.370474 13620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:06.371021 13620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:06.372176 13620 hybrid_clock.cc:648] HybridClock initialized: now 1790639406372156 us; error 30 us; skew 500 ppm
Sep 28 23:50:06 dist-test-slave-p5bw krb5kdc[12817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639406, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:06.374814 13620 init.cc:377] Logged in from keytab as oryx/127.6.152.1@KRBTEST.COM (short username oryx)
I20260928 23:50:06.375643 13620 webserver.cc:460] Webserver started at http://127.6.152.1:36617/ using document root <none> and password file <none>
I20260928 23:50:06.376180 13620 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:06.376292 13620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:06.379913 13620 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:50:06.381953 13724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:06.383119 13620 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:50:06.383248 13620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "af5a3b2b50fa4554a12490c58ccbb825"
format_stamp: "Formatted at 2026-09-28 23:50:05 on dist-test-slave-p5bw"
server_key: "33f0f2275a25a9663354ffdbe899003a"
server_key_iv: "39dd5928d35f9b65630fc4baf760314f"
server_key_version: "encryptionkey@0"
I20260928 23:50:06.383774 13620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:06.406418 13620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:06.409602 13620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:06.409814 13620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:06.410424 13620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:50:06.411268 13620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:06.411339 13620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:06.411414 13620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:06.411456 13620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:06.420153 13620 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:38171
I20260928 23:50:06.420486 13837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:38171 every 8 connection(s)
I20260928 23:50:06.421304 13620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:50:06.422086 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 13620
I20260928 23:50:06.422279 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 13059
I20260928 23:50:06.430639 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:46775
--local_ip_for_outbound_sockets=127.6.152.2
--tserver_master_addrs=127.6.152.62:36193
--webserver_port=44809
--webserver_interface=127.6.152.2
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.2.keytab
--principal=oryx/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:38547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.152.62:36193,127.6.152.61:46027
--tserver_master_addrs=127.6.152.62:36193,127.6.152.61:46027,127.6.152.60:34469 with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:50:06 dist-test-slave-p5bw krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639406, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM
Sep 28 23:50:06 dist-test-slave-p5bw krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639406, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.61@KRBTEST.COM
Sep 28 23:50:06 dist-test-slave-p5bw krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639406, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.60@KRBTEST.COM
I20260928 23:50:06.440316 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.423336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:41437 (local address 127.6.152.61:46027)
0928 23:50:06.425983 (+ 2647us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.425986 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.425999 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:50:06.426662 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.426669 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.426738 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.426794 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:06.428798 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.429643 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.431701 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.432047 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.432132 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.435988 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:06.435993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:06.435995 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:06.436025 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:06.437767 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.438409 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.438415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.438432 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.438518 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.438969 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.438974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.438977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.439206 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:06.439303 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.440004 (+ 701us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.440134 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2560}
I20260928 23:50:06.440418 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.423392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:60417 (local address 127.6.152.62:36193)
0928 23:50:06.423753 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.423759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.425204 (+ 1445us) server_negotiation.cc:407] Connection header received
0928 23:50:06.427155 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.427161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.427225 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.427282 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:06.428797 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.429718 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.431733 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.432115 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.432209 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.435996 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:06.436002 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:06.436006 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:06.436041 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:06.438016 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.438580 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.438588 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.438591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.438690 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.439046 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.439053 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.439058 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.439329 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:06.439417 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.440052 (+ 635us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.440200 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264}
I20260928 23:50:06.441509 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.423515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:41781 (local address 127.6.152.60:34469)
0928 23:50:06.424729 (+ 1214us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.424736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.425731 (+ 995us) server_negotiation.cc:407] Connection header received
0928 23:50:06.426813 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.426819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.426883 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.426973 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:06.428793 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.429870 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.431701 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.432054 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.432149 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.437552 (+ 5403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:06.437558 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:06.437561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:06.437592 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:06.439373 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.439807 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.439810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.439812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.439896 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.440195 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.440198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.440200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.440452 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:06.440540 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.441194 (+ 654us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.441308 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1119,"thread_start_us":81,"threads_started":1}
I20260928 23:50:06.441948 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.423663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.6.152.1:60417)
0928 23:50:06.425029 (+ 1366us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:06.425096 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:06.426416 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:06.427436 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:06.427441 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:06.427786 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:06.428625 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.428648 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.429866 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.429870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.431589 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.431603 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.432394 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.432399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.432621 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.433513 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:06.433533 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:06.435779 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:06.438168 (+ 2389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.438174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.438179 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.438451 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.438821 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.438825 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.438829 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.438928 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.439504 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:06.439509 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:06.439816 (+ 307us) client_negotiation.cc:769] Sending connection context
0928 23:50:06.441164 (+ 1348us) client_negotiation.cc:240] Negotiation successful
0928 23:50:06.441325 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1146,"mutex_wait_us":290,"spinlock_wait_cycles":27648,"thread_start_us":130,"threads_started":1}
I20260928 23:50:06.442162 13845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.423822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:34469 (local address 127.6.152.1:41781)
0928 23:50:06.425647 (+ 1825us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:06.425658 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:06.426613 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:06.427133 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:06.427143 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:06.427791 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:06.428639 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.428646 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.430008 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.430011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.431566 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.431576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.432303 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.432307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.432624 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.433601 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:06.433618 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:06.437385 (+ 3767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:06.439483 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.439485 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.439487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.439699 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.440007 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.440012 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.440014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.440095 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.440579 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:06.440582 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:06.440673 (+ 91us) client_negotiation.cc:769] Sending connection context
0928 23:50:06.440767 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 23:50:06.440893 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1768,"mutex_wait_us":560,"spinlock_wait_cycles":19072,"thread_start_us":132,"threads_started":1}
I20260928 23:50:06.443146 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.423664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:46027 (local address 127.6.152.1:41437)
0928 23:50:06.425241 (+ 1577us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:06.425257 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:06.426428 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:06.427134 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:06.427143 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:06.427788 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:06.428627 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.428646 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.429803 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.429806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.431567 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.431576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.432303 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.432307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.432629 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.433448 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:06.433479 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:06.435807 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:06.437900 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.437908 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.437924 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.438290 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.438648 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.438654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.438658 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.438850 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.439388 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:06.439396 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:06.439722 (+ 326us) client_negotiation.cc:769] Sending connection context
0928 23:50:06.440477 (+ 755us) client_negotiation.cc:240] Negotiation successful
0928 23:50:06.440834 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1470,"mutex_wait_us":559,"spinlock_wait_cycles":167808,"thread_start_us":151,"threads_started":1}
I20260928 23:50:06.443504 13839 heartbeater.cc:347] Connected to a master server at 127.6.152.62:36193
I20260928 23:50:06.443725 13839 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:06.444134 13840 heartbeater.cc:347] Connected to a master server at 127.6.152.61:46027
I20260928 23:50:06.444216 13840 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:06.444291 13839 heartbeater.cc:511] Master 127.6.152.62:36193 requested a full tablet report, sending...
I20260928 23:50:06.444363 13840 heartbeater.cc:511] Master 127.6.152.61:46027 requested a full tablet report, sending...
I20260928 23:50:06.444829 13838 heartbeater.cc:347] Connected to a master server at 127.6.152.60:34469
I20260928 23:50:06.444917 13838 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:06.445070 13838 heartbeater.cc:511] Master 127.6.152.60:34469 requested a full tablet report, sending...
I20260928 23:50:06.445844 13466 ts_manager.cc:194] Registered new tserver with Master: af5a3b2b50fa4554a12490c58ccbb825 (127.6.152.1:38171)
I20260928 23:50:06.445959 13659 ts_manager.cc:194] Registered new tserver with Master: af5a3b2b50fa4554a12490c58ccbb825 (127.6.152.1:38171)
I20260928 23:50:06.446522 12862 ts_manager.cc:194] Re-registered known tserver with Master: af5a3b2b50fa4554a12490c58ccbb825 (127.6.152.1:38171)
I20260928 23:50:06.447250 12862 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.152.1@KRBTEST.COM'} at 127.6.152.1:60417
W20260928 23:50:06.545744 13846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:06.546018 13846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:06.546097 13846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:06.550251 13846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:06.550323 13846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:06.550414 13846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:50:06.555384 13846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:38547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.2.keytab
--principal=oryx/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:46775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=44809
--tserver_master_addrs=127.6.152.62:36193,127.6.152.61:46027,127.6.152.60:34469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:06.556365 13846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:06.557464 13846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:06.566913 13854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:06.566947 13851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:06.566915 13852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:06.567126 13846 server_base.cc:1034] running on GCE node
I20260928 23:50:06.567696 13846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:06.568332 13846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:06.569533 13846 hybrid_clock.cc:648] HybridClock initialized: now 1790639406569504 us; error 40 us; skew 500 ppm
Sep 28 23:50:06 dist-test-slave-p5bw krb5kdc[12817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639406, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:06.572144 13846 init.cc:377] Logged in from keytab as oryx/127.6.152.2@KRBTEST.COM (short username oryx)
I20260928 23:50:06.573015 13846 webserver.cc:460] Webserver started at http://127.6.152.2:44809/ using document root <none> and password file <none>
I20260928 23:50:06.573506 13846 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:06.573576 13846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:06.577375 13846 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:50:06.578411 13325 heartbeater.cc:503] Master 127.6.152.62:36193 was elected leader, sending a full tablet report...
I20260928 23:50:06.579432 13861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:06.580862 13846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:50:06.581015 13846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "90b865839cc84946afb5399f87beee52"
format_stamp: "Formatted at 2026-09-28 23:50:05 on dist-test-slave-p5bw"
server_key: "975d731c0fa135483466bc0932380d62"
server_key_iv: "7302b6613e53fb1be3a135eb0ef62ae8"
server_key_version: "encryptionkey@0"
I20260928 23:50:06.581514 13846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:06.617389 13846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:06.620622 13846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:06.620811 13846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:06.621452 13846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:50:06.622294 13846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:06.622339 13846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:06.622401 13846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:06.622433 13846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:06.631140 13846 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:46775
I20260928 23:50:06.631166 13975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:46775 every 8 connection(s)
I20260928 23:50:06.632182 13846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:50:06.638383 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 13846
I20260928 23:50:06.638564 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 13195
Sep 28 23:50:06 dist-test-slave-p5bw krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639406, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.60@KRBTEST.COM
Sep 28 23:50:06 dist-test-slave-p5bw krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639406, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.61@KRBTEST.COM
Sep 28 23:50:06 dist-test-slave-p5bw krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639406, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM
I20260928 23:50:06.649361 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:33437
--local_ip_for_outbound_sockets=127.6.152.3
--tserver_master_addrs=127.6.152.62:36193
--webserver_port=36631
--webserver_interface=127.6.152.3
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.3.keytab
--principal=oryx/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:38547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.152.62:36193,127.6.152.61:46027
--tserver_master_addrs=127.6.152.62:36193,127.6.152.61:46027,127.6.152.60:34469 with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:50:06.652590 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.634059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:52647 (local address 127.6.152.60:34469)
0928 23:50:06.634189 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.634193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.635510 (+ 1317us) server_negotiation.cc:407] Connection header received
0928 23:50:06.636425 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.636428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.636486 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.636538 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:06.638401 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.639271 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.641370 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.641834 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.641937 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.647285 (+ 5348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:06.647289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:06.647292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:06.647323 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:06.649393 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.650453 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.650456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.650457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.650552 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.651198 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.651201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.651203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.651446 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:06.651523 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.652296 (+ 773us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.652432 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:50:06.652590 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.634032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:51165 (local address 127.6.152.62:36193)
0928 23:50:06.634199 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.634204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.635321 (+ 1117us) server_negotiation.cc:407] Connection header received
0928 23:50:06.636399 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.636402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.636470 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.636567 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:06.638371 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.639199 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.641342 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.641691 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.641768 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.647285 (+ 5517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:06.647289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:06.647291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:06.647321 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:06.649682 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.650448 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.650451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.650453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.650727 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.651198 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.651201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.651203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.651438 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:06.651523 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.652296 (+ 773us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.652420 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 23:50:06.652689 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.634811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:40417 (local address 127.6.152.61:46027)
0928 23:50:06.634922 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.634926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.635819 (+ 893us) server_negotiation.cc:407] Connection header received
0928 23:50:06.636698 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.636702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.636755 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.636860 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:06.638370 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.639199 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.641341 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.641688 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.641768 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.648131 (+ 6363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:06.648138 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:06.648144 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:06.648180 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:06.650449 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.651077 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.651083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.651085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.651177 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.651503 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.651508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.651511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.651750 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:06.651825 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.652297 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.652446 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:50:06.654441 13982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.634443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:34469 (local address 127.6.152.2:52647)
0928 23:50:06.635405 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:06.635416 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:06.636178 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:06.636763 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:06.636770 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:06.637569 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:06.638208 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.638221 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.639441 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.639447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.641216 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.641230 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.642130 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.642137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.642508 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.643492 (+ 984us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:06.643517 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:06.647069 (+ 3552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:06.649806 (+ 2737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.649813 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.649833 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.650327 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.650851 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.650854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.650857 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.651084 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.651587 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:06.651594 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:06.651997 (+ 403us) client_negotiation.cc:769] Sending connection context
0928 23:50:06.653306 (+ 1309us) client_negotiation.cc:240] Negotiation successful
0928 23:50:06.653486 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":456,"spinlock_wait_cycles":19584,"thread_start_us":141,"threads_started":1}
I20260928 23:50:06.654433 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.634685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:46027 (local address 127.6.152.2:40417)
0928 23:50:06.635721 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:06.635731 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:06.636161 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:06.636904 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:06.636909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:06.637534 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:06.638194 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.638214 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.639384 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.639388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.641203 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.641208 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.641983 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.641986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.642459 (+ 473us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.643607 (+ 1148us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:06.643623 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:06.647947 (+ 4324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:06.650661 (+ 2714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.650667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.650671 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.650945 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.651292 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.651296 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.651300 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.651401 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.651911 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:06.651914 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:06.652037 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 23:50:06.652911 (+ 874us) client_negotiation.cc:240] Negotiation successful
0928 23:50:06.653425 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"mutex_wait_us":710,"spinlock_wait_cycles":48256,"thread_start_us":78,"threads_started":1}
I20260928 23:50:06.654433 13981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.634443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.6.152.2:51165)
0928 23:50:06.635176 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:06.635228 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:06.636161 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:06.636762 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:06.636769 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:06.637531 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:06.638193 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.638214 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.639384 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.639388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.641196 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.641207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.641983 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.641986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.642470 (+ 484us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.643643 (+ 1173us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:06.643654 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:06.647069 (+ 3415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:06.649806 (+ 2737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.649813 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.649833 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.650327 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.650852 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.650855 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.650857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.651085 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.651587 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:06.651594 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:06.651997 (+ 403us) client_negotiation.cc:769] Sending connection context
0928 23:50:06.652838 (+ 841us) client_negotiation.cc:240] Negotiation successful
0928 23:50:06.653303 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":485,"spinlock_wait_cycles":50048,"thread_start_us":145,"threads_started":1}
I20260928 23:50:06.655786 13978 heartbeater.cc:347] Connected to a master server at 127.6.152.61:46027
I20260928 23:50:06.656066 13978 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:06.656790 13978 heartbeater.cc:511] Master 127.6.152.61:46027 requested a full tablet report, sending...
I20260928 23:50:06.655783 13977 heartbeater.cc:347] Connected to a master server at 127.6.152.62:36193
I20260928 23:50:06.657033 13977 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:06.657192 13977 heartbeater.cc:511] Master 127.6.152.62:36193 requested a full tablet report, sending...
I20260928 23:50:06.655805 13976 heartbeater.cc:347] Connected to a master server at 127.6.152.60:34469
I20260928 23:50:06.657505 13976 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:06.657668 13976 heartbeater.cc:511] Master 127.6.152.60:34469 requested a full tablet report, sending...
I20260928 23:50:06.658038 12862 ts_manager.cc:194] Re-registered known tserver with Master: 90b865839cc84946afb5399f87beee52 (127.6.152.2:46775)
I20260928 23:50:06.658171 13466 ts_manager.cc:194] Registered new tserver with Master: 90b865839cc84946afb5399f87beee52 (127.6.152.2:46775)
I20260928 23:50:06.658257 13659 ts_manager.cc:194] Registered new tserver with Master: 90b865839cc84946afb5399f87beee52 (127.6.152.2:46775)
I20260928 23:50:06.658732 12862 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.152.2@KRBTEST.COM'} at 127.6.152.2:51165
W20260928 23:50:06.762497 13983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:06.762753 13983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:06.762795 13983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:06.766878 13983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:06.766968 13983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:06.767076 13983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:50:06.772320 13983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:38547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.3.keytab
--principal=oryx/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:33437
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=36631
--tserver_master_addrs=127.6.152.62:36193,127.6.152.61:46027,127.6.152.60:34469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:06.773440 13983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:06.774430 13983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:06.784148 13988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:06.784145 13989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:06.784142 13991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:06.784591 13983 server_base.cc:1034] running on GCE node
I20260928 23:50:06.785065 13983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:06.785750 13983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:06.786932 13983 hybrid_clock.cc:648] HybridClock initialized: now 1790639406786921 us; error 36 us; skew 500 ppm
Sep 28 23:50:06 dist-test-slave-p5bw krb5kdc[12817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639406, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:06.789800 13983 init.cc:377] Logged in from keytab as oryx/127.6.152.3@KRBTEST.COM (short username oryx)
I20260928 23:50:06.790652 13983 webserver.cc:460] Webserver started at http://127.6.152.3:36631/ using document root <none> and password file <none>
I20260928 23:50:06.791249 13983 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:06.791345 13983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:06.795126 13983 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 23:50:06.797261 13998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:06.798511 13983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:50:06.798686 13983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "24026d9eea29427bb69abcf470ecf1da"
format_stamp: "Formatted at 2026-09-28 23:50:05 on dist-test-slave-p5bw"
server_key: "a328a4cc236430af75561d1be529cd85"
server_key_iv: "c88b0fc4e4388bd96850b5e1d7febcb5"
server_key_version: "encryptionkey@0"
I20260928 23:50:06.799264 13983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:06.808174 14004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.804427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:34469 (local address 127.0.0.1:39192)
0928 23:50:06.804710 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:06.804722 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:06.805052 (+ 330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:06.805355 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:06.805359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:06.805379 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:06.805490 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.805495 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.806284 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.806287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.807073 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.807082 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.807744 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.807747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.807835 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.807859 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 23:50:06.807957 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 23:50:06.808038 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":60,"threads_started":1}
I20260928 23:50:06.808171 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.804544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39192 (local address 127.6.152.60:34469)
0928 23:50:06.804675 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.804680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.804797 (+ 117us) server_negotiation.cc:407] Connection header received
0928 23:50:06.805183 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.805187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.805245 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.805341 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:06.805597 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.806174 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.807193 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.807620 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.807740 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.807884 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.807970 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.808038 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:50:06.808898 13663 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 23:50:06.809327 13698 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b20d904b67749e9a3276bfdd467c7be. Latest consensus state: current_term: 1 leader_uuid: "8b20d904b67749e9a3276bfdd467c7be" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: NON_VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: true } } }
I20260928 23:50:06.809461 13698 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [sys.catalog]: This master's current role is: LEARNER
I20260928 23:50:06.812189 13983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:06.815018 13983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:06.815217 13983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:06.815784 13983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:50:06.816602 13983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:06.816671 13983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:06.816742 13983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:06.816787 13983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:06.826354 13983 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:33437
I20260928 23:50:06.826387 14113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:33437 every 8 connection(s)
I20260928 23:50:06.827509 13983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:50:06.836549 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 13983
Sep 28 23:50:06 dist-test-slave-p5bw krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639406, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM
Sep 28 23:50:06 dist-test-slave-p5bw krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639406, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.61@KRBTEST.COM
Sep 28 23:50:06 dist-test-slave-p5bw krb5kdc[12817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639406, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.60@KRBTEST.COM
I20260928 23:50:06.846946 14120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.829807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:46027 (local address 127.6.152.3:54947)
0928 23:50:06.831020 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:06.831085 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:06.831930 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:06.832509 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:06.832517 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:06.833131 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:06.833859 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.833875 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.834968 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.834971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.836553 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.836562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.837487 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.837491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.837753 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.838496 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:06.838519 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:06.841498 (+ 2979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:06.843485 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.843491 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.843509 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.843828 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.844153 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.844156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.844158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.844342 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.844876 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:06.844884 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:06.845230 (+ 346us) client_negotiation.cc:769] Sending connection context
0928 23:50:06.845447 (+ 217us) client_negotiation.cc:240] Negotiation successful
0928 23:50:06.845790 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1008,"mutex_wait_us":377,"spinlock_wait_cycles":7808,"thread_start_us":127,"threads_started":1}
I20260928 23:50:06.847318 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.830873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:51653 (local address 127.6.152.60:34469)
0928 23:50:06.830997 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.831003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.832363 (+ 1360us) server_negotiation.cc:407] Connection header received
0928 23:50:06.832520 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.832524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.832569 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.832621 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:06.834017 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.834824 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.838728 (+ 3904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.839080 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.839178 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.842694 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:06.842699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:06.842702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:06.842729 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:06.844697 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.845497 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.845502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.845505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.845593 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.845917 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.845921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.845922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.846136 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:06.846224 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.847050 (+ 826us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.847177 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:50:06.847838 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.829459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:54947 (local address 127.6.152.61:46027)
0928 23:50:06.829628 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.829635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.831232 (+ 1597us) server_negotiation.cc:407] Connection header received
0928 23:50:06.832120 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.832124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.832176 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.832224 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:06.834012 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.834811 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.836681 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.837097 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.837170 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.841673 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:06.841676 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:06.841679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:06.841704 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:06.843350 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.843943 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.843950 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.843954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.844037 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.844457 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.844461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.844463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.844705 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:06.844796 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.847527 (+ 2731us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.847650 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":63}
I20260928 23:50:06.846949 14119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.830012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:34469 (local address 127.6.152.3:51653)
0928 23:50:06.832288 (+ 2276us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:06.832300 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:06.832419 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:06.832787 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:06.832790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:06.833122 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:06.833870 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.833877 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.834970 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.834974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.838590 (+ 3616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.838600 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.839365 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.839369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.839461 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.839926 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:06.839945 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:06.842302 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:06.844829 (+ 2527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.844832 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.844834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.845374 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.845710 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.845713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.845717 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.845811 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.846215 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:06.846219 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:06.846316 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 23:50:06.846414 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 23:50:06.846539 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2222,"mutex_wait_us":295,"spinlock_wait_cycles":9088,"thread_start_us":109,"threads_started":1}
I20260928 23:50:06.849081 14116 heartbeater.cc:347] Connected to a master server at 127.6.152.61:46027
I20260928 23:50:06.849324 14116 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:06.849906 14116 heartbeater.cc:511] Master 127.6.152.61:46027 requested a full tablet report, sending...
I20260928 23:50:06.851132 13466 ts_manager.cc:194] Registered new tserver with Master: 24026d9eea29427bb69abcf470ecf1da (127.6.152.3:33437)
I20260928 23:50:06.851568 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.838593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53120 (local address 127.6.152.62:36193)
0928 23:50:06.839361 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.839368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.839626 (+ 258us) server_negotiation.cc:407] Connection header received
0928 23:50:06.840321 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.840325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.840374 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.840456 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:06.841423 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.842212 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.843533 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.843906 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.844006 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.845771 (+ 1765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:06.845775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:06.845777 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:06.845803 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:06.849333 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.849885 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.849889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.849890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.850014 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.850339 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.850343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.850344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.850560 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:06.851193 (+ 633us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.851270 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.851406 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":61,"threads_started":1}
I20260928 23:50:06.852350 14114 heartbeater.cc:347] Connected to a master server at 127.6.152.60:34469
I20260928 23:50:06.852492 14114 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:06.852692 14114 heartbeater.cc:511] Master 127.6.152.60:34469 requested a full tablet report, sending...
I20260928 23:50:06.852720 14126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.838989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52680 (local address 127.6.152.61:46027)
0928 23:50:06.840420 (+ 1431us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.840427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.840827 (+ 400us) server_negotiation.cc:407] Connection header received
0928 23:50:06.840881 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.840884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.840933 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.841019 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:06.841995 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.842789 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.844114 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.844510 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.848416 (+ 3906us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.848528 (+ 112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:06.848533 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:06.848536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:06.848555 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:06.850364 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.850844 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.850850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.850851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.850936 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.851375 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.851379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.851381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.851597 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:06.851711 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.852418 (+ 707us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.852534 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3794,"server-negotiator.queue_time_us":1356,"thread_start_us":67,"threads_started":1}
I20260928 23:50:06.853825 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.830813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:58711 (local address 127.6.152.62:36193)
0928 23:50:06.830956 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.830961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.831190 (+ 229us) server_negotiation.cc:407] Connection header received
0928 23:50:06.832103 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.832106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.832161 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.832237 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:06.834017 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.834824 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.836800 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.837147 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.837224 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.846657 (+ 9433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:06.846661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:06.846664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:06.846688 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:06.848258 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.850693 (+ 2435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.850699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.850701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.850789 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.851919 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.851925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.851930 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.852174 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:06.852266 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.853560 (+ 1294us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.853688 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":64}
I20260928 23:50:06.853971 13659 ts_manager.cc:194] Registered new tserver with Master: 24026d9eea29427bb69abcf470ecf1da (127.6.152.3:33437)
I20260928 23:50:06.855298 14118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.829807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.6.152.3:58711)
0928 23:50:06.831096 (+ 1289us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:06.831107 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:06.831906 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:06.832508 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:06.832516 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:06.833122 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:06.833859 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.833877 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.834970 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.834974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.836558 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.836570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.837430 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.837437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.837757 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.838603 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:06.838618 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:06.846520 (+ 7902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:06.849900 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.849903 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.849905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.850151 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.851503 (+ 1352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:06.851506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:06.851508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:06.851598 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:06.852439 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:06.852445 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:06.852535 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 23:50:06.854976 (+ 2441us) client_negotiation.cc:240] Negotiation successful
0928 23:50:06.855117 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1207,"mutex_wait_us":408,"spinlock_wait_cycles":96128,"thread_start_us":144,"threads_started":1}
I20260928 23:50:06.856215 14115 heartbeater.cc:347] Connected to a master server at 127.6.152.62:36193
I20260928 23:50:06.856307 14115 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:06.856505 14115 heartbeater.cc:511] Master 127.6.152.62:36193 requested a full tablet report, sending...
I20260928 23:50:06.857163 12861 ts_manager.cc:194] Re-registered known tserver with Master: 24026d9eea29427bb69abcf470ecf1da (127.6.152.3:33437)
I20260928 23:50:06.854473 12862 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 23:50:06.857825 12861 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.152.3@KRBTEST.COM'} at 127.6.152.3:58711
W20260928 23:50:06.858294 12862 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:50:06.860775 14129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.839870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39208 (local address 127.6.152.60:34469)
0928 23:50:06.840125 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.840129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.844520 (+ 4391us) server_negotiation.cc:407] Connection header received
0928 23:50:06.850289 (+ 5769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.850296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.850350 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.850413 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:06.852698 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.853377 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.854960 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.855303 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.855412 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.857221 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:06.857224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:06.857227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:06.857251 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:06.858886 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.859527 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.859531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.859533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.859621 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:06.859913 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:06.859916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:06.859918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:06.860145 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:06.860276 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.860472 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.860590 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":118,"threads_started":1}
I20260928 23:50:06.861133 13999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b20d904b67749e9a3276bfdd467c7be" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: NON_VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: true } } }
I20260928 23:50:06.861276 13999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:06.865370 13999 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be: attempting to promote NON_VOTER ec71e3446d104e96b5462e085b9e757b to VOTER
I20260928 23:50:06.865731 13999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: false } }
I20260928 23:50:06.866822 13663 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [term 1 LEARNER]: Refusing update from remote peer 8b20d904b67749e9a3276bfdd467c7be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 23:50:06.867326 13484 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [term 1 FOLLOWER]: Refusing update from remote peer 8b20d904b67749e9a3276bfdd467c7be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260928 23:50:06.867300 13999 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 23:50:06.867643 13999 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 23:50:06.869213 14133 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ec71e3446d104e96b5462e085b9e757b (127.6.152.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: false } } }
I20260928 23:50:06.869932 13663 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ec71e3446d104e96b5462e085b9e757b (127.6.152.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: false } } }
I20260928 23:50:06.870791 14133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8b20d904b67749e9a3276bfdd467c7be" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: false } } }
I20260928 23:50:06.870906 14133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:06.871049 14133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b20d904b67749e9a3276bfdd467c7be" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: false } } }
I20260928 23:50:06.871141 14133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:06.871116 14132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b20d904b67749e9a3276bfdd467c7be" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: false } } }
I20260928 23:50:06.871323 14132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:06.871481 13484 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ec71e3446d104e96b5462e085b9e757b (127.6.152.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: false } } }
I20260928 23:50:06.875465 13698 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b20d904b67749e9a3276bfdd467c7be" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: false } } }
I20260928 23:50:06.875602 13698 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:50:06.877143 14141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b20d904b67749e9a3276bfdd467c7be" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: false } } }
I20260928 23:50:06.877283 14141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:50:06.879014 14145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.873679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:46775 (local address 127.0.0.1:58822)
0928 23:50:06.873902 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:06.873916 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:06.874007 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:06.874750 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:06.874753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:06.874771 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:06.874903 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.874909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.876124 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.876128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.877369 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.877381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.878512 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.878516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.878682 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.878709 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 23:50:06.878780 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 23:50:06.878842 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":32,"thread_start_us":87,"threads_started":1}
I20260928 23:50:06.879228 14146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.873911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58822 (local address 127.6.152.2:46775)
0928 23:50:06.874232 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.874260 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.874280 (+ 20us) server_negotiation.cc:407] Connection header received
0928 23:50:06.874345 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.874349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.874599 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.874791 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:06.875016 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.876018 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.877518 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.878379 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.878523 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.878674 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.878826 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.878923 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":126,"threads_started":1}
I20260928 23:50:06.880434 14143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.873365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:33437 (local address 127.0.0.1:52414)
0928 23:50:06.873625 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:06.873637 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:06.873719 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:06.876391 (+ 2672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:06.876394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:06.876410 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:06.876487 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.876492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.877788 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.877792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.878939 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.878948 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.880067 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.880071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.880156 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.880178 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:50:06.880238 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 23:50:06.880295 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":50,"thread_start_us":109,"threads_started":1}
I20260928 23:50:06.881017 14144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.873569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52414 (local address 127.6.152.3:33437)
0928 23:50:06.875944 (+ 2375us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.875953 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.875972 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:50:06.876034 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.876038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.876254 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.876424 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:06.876559 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.877664 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.879067 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.879888 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.880557 (+ 669us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.880698 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.880745 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.880852 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2281,"thread_start_us":99,"threads_started":1}
I20260928 23:50:06.882349 14004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.872733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:38171 (local address 127.0.0.1:50564)
0928 23:50:06.872918 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:06.872933 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:06.873014 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:06.878058 (+ 5044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:06.878061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:06.878079 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:06.878156 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.878162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.879738 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.879742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.880860 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:06.880880 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.882027 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.882031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:06.882116 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.882137 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 23:50:06.882184 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 23:50:06.882247 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260928 23:50:06.882385 13909 tablet_service.cc:1467] Processing CreateTablet for tablet f1266a5341424db092b36df8cfcc76cc (DEFAULT_TABLE table=test-table [id=ca90cac40f1e4782b487dd47cecd94df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:50:06.883108 14142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:06.872906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50564 (local address 127.6.152.1:38171)
0928 23:50:06.877591 (+ 4685us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:06.877603 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:06.877625 (+ 22us) server_negotiation.cc:407] Connection header received
0928 23:50:06.877694 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:06.877700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:06.877879 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:06.878304 (+ 425us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:06.878450 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.879604 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.880990 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:06.881901 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:06.882691 (+ 790us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:06.882827 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:06.882896 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 23:50:06.882986 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4577,"thread_start_us":156,"threads_started":1}
I20260928 23:50:06.883556 13909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1266a5341424db092b36df8cfcc76cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:06.883659 14048 tablet_service.cc:1467] Processing CreateTablet for tablet f1266a5341424db092b36df8cfcc76cc (DEFAULT_TABLE table=test-table [id=ca90cac40f1e4782b487dd47cecd94df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:50:06.884657 14048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1266a5341424db092b36df8cfcc76cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:06.885656 13772 tablet_service.cc:1467] Processing CreateTablet for tablet f1266a5341424db092b36df8cfcc76cc (DEFAULT_TABLE table=test-table [id=ca90cac40f1e4782b487dd47cecd94df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:50:06.886567 13772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1266a5341424db092b36df8cfcc76cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:06.890563 14147 tablet_bootstrap.cc:492] T f1266a5341424db092b36df8cfcc76cc P 90b865839cc84946afb5399f87beee52: Bootstrap starting.
I20260928 23:50:06.892205 14148 tablet_bootstrap.cc:492] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da: Bootstrap starting.
I20260928 23:50:06.892841 14147 tablet_bootstrap.cc:654] T f1266a5341424db092b36df8cfcc76cc P 90b865839cc84946afb5399f87beee52: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:06.893709 14147 log.cc:824] T f1266a5341424db092b36df8cfcc76cc P 90b865839cc84946afb5399f87beee52: Log is configured to *not* fsync() on all Append() calls
I20260928 23:50:06.894215 14149 tablet_bootstrap.cc:492] T f1266a5341424db092b36df8cfcc76cc P af5a3b2b50fa4554a12490c58ccbb825: Bootstrap starting.
I20260928 23:50:06.894361 14148 tablet_bootstrap.cc:654] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:06.895201 14148 log.cc:824] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da: Log is configured to *not* fsync() on all Append() calls
I20260928 23:50:06.896106 14149 tablet_bootstrap.cc:654] T f1266a5341424db092b36df8cfcc76cc P af5a3b2b50fa4554a12490c58ccbb825: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:06.896728 14149 log.cc:824] T f1266a5341424db092b36df8cfcc76cc P af5a3b2b50fa4554a12490c58ccbb825: Log is configured to *not* fsync() on all Append() calls
I20260928 23:50:06.897116 14148 tablet_bootstrap.cc:492] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da: No bootstrap required, opened a new log
I20260928 23:50:06.897325 14148 ts_tablet_manager.cc:1397] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 23:50:06.898124 14147 tablet_bootstrap.cc:492] T f1266a5341424db092b36df8cfcc76cc P 90b865839cc84946afb5399f87beee52: No bootstrap required, opened a new log
I20260928 23:50:06.898308 14147 ts_tablet_manager.cc:1397] T f1266a5341424db092b36df8cfcc76cc P 90b865839cc84946afb5399f87beee52: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260928 23:50:06.898543 14149 tablet_bootstrap.cc:492] T f1266a5341424db092b36df8cfcc76cc P af5a3b2b50fa4554a12490c58ccbb825: No bootstrap required, opened a new log
I20260928 23:50:06.898706 14149 ts_tablet_manager.cc:1397] T f1266a5341424db092b36df8cfcc76cc P af5a3b2b50fa4554a12490c58ccbb825: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 23:50:06.901120 14148 raft_consensus.cc:348] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5a3b2b50fa4554a12490c58ccbb825" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 38171 } } peers { permanent_uuid: "24026d9eea29427bb69abcf470ecf1da" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33437 } } peers { permanent_uuid: "90b865839cc84946afb5399f87beee52" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46775 } }
I20260928 23:50:06.901361 14148 raft_consensus.cc:374] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:06.901412 14148 raft_consensus.cc:729] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24026d9eea29427bb69abcf470ecf1da, State: Initialized, Role: FOLLOWER
I20260928 23:50:06.901865 14148 consensus_queue.cc:260] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5a3b2b50fa4554a12490c58ccbb825" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 38171 } } peers { permanent_uuid: "24026d9eea29427bb69abcf470ecf1da" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33437 } } peers { permanent_uuid: "90b865839cc84946afb5399f87beee52" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46775 } }
I20260928 23:50:06.902912 14148 ts_tablet_manager.cc:1428] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:50:06.902894 14147 raft_consensus.cc:348] T f1266a5341424db092b36df8cfcc76cc P 90b865839cc84946afb5399f87beee52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5a3b2b50fa4554a12490c58ccbb825" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 38171 } } peers { permanent_uuid: "24026d9eea29427bb69abcf470ecf1da" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33437 } } peers { permanent_uuid: "90b865839cc84946afb5399f87beee52" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46775 } }
I20260928 23:50:06.903019 14149 raft_consensus.cc:348] T f1266a5341424db092b36df8cfcc76cc P af5a3b2b50fa4554a12490c58ccbb825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5a3b2b50fa4554a12490c58ccbb825" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 38171 } } peers { permanent_uuid: "24026d9eea29427bb69abcf470ecf1da" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33437 } } peers { permanent_uuid: "90b865839cc84946afb5399f87beee52" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46775 } }
I20260928 23:50:06.903149 14147 raft_consensus.cc:374] T f1266a5341424db092b36df8cfcc76cc P 90b865839cc84946afb5399f87beee52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:06.903226 14149 raft_consensus.cc:374] T f1266a5341424db092b36df8cfcc76cc P af5a3b2b50fa4554a12490c58ccbb825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:06.903280 14149 raft_consensus.cc:729] T f1266a5341424db092b36df8cfcc76cc P af5a3b2b50fa4554a12490c58ccbb825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af5a3b2b50fa4554a12490c58ccbb825, State: Initialized, Role: FOLLOWER
I20260928 23:50:06.903280 14147 raft_consensus.cc:729] T f1266a5341424db092b36df8cfcc76cc P 90b865839cc84946afb5399f87beee52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b865839cc84946afb5399f87beee52, State: Initialized, Role: FOLLOWER
I20260928 23:50:06.903292 14115 heartbeater.cc:503] Master 127.6.152.62:36193 was elected leader, sending a full tablet report...
I20260928 23:50:06.903772 14149 consensus_queue.cc:260] T f1266a5341424db092b36df8cfcc76cc P af5a3b2b50fa4554a12490c58ccbb825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5a3b2b50fa4554a12490c58ccbb825" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 38171 } } peers { permanent_uuid: "24026d9eea29427bb69abcf470ecf1da" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33437 } } peers { permanent_uuid: "90b865839cc84946afb5399f87beee52" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46775 } }
I20260928 23:50:06.903756 14147 consensus_queue.cc:260] T f1266a5341424db092b36df8cfcc76cc P 90b865839cc84946afb5399f87beee52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5a3b2b50fa4554a12490c58ccbb825" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 38171 } } peers { permanent_uuid: "24026d9eea29427bb69abcf470ecf1da" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33437 } } peers { permanent_uuid: "90b865839cc84946afb5399f87beee52" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46775 } }
I20260928 23:50:06.904691 14147 ts_tablet_manager.cc:1428] T f1266a5341424db092b36df8cfcc76cc P 90b865839cc84946afb5399f87beee52: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 23:50:06.905186 13977 heartbeater.cc:503] Master 127.6.152.62:36193 was elected leader, sending a full tablet report...
I20260928 23:50:06.905556 14149 ts_tablet_manager.cc:1428] T f1266a5341424db092b36df8cfcc76cc P af5a3b2b50fa4554a12490c58ccbb825: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 23:50:06.905923 13839 heartbeater.cc:503] Master 127.6.152.62:36193 was elected leader, sending a full tablet report...
W20260928 23:50:06.925405 13841 tablet.cc:2369] T f1266a5341424db092b36df8cfcc76cc P af5a3b2b50fa4554a12490c58ccbb825: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:50:07.079447 14117 tablet.cc:2369] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:50:07.092756 14153 raft_consensus.cc:482] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:50:07.093169 14153 raft_consensus.cc:504] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5a3b2b50fa4554a12490c58ccbb825" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 38171 } } peers { permanent_uuid: "24026d9eea29427bb69abcf470ecf1da" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33437 } } peers { permanent_uuid: "90b865839cc84946afb5399f87beee52" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46775 } }
I20260928 23:50:07.094365 14153 leader_election.cc:288] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af5a3b2b50fa4554a12490c58ccbb825 (127.6.152.1:38171), 90b865839cc84946afb5399f87beee52 (127.6.152.2:46775)
I20260928 23:50:07.098635 14146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:07.094874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:38363 (local address 127.6.152.2:46775)
0928 23:50:07.095020 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:07.095024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:07.095164 (+ 140us) server_negotiation.cc:407] Connection header received
0928 23:50:07.095297 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:07.095300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:07.095350 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:07.095418 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:07.095786 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:07.096264 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:07.097509 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:07.097998 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:07.098079 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:07.098182 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:07.098428 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 23:50:07.098503 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 23:50:07.098652 14118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:07.094663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:38171 (local address 127.6.152.3:34573)
0928 23:50:07.094808 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:07.094826 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:07.094987 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:07.095303 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:07.095305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:07.095319 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:07.095464 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:07.095471 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:07.096409 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:07.096412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:07.097436 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:07.097463 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:07.098201 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:07.098205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:07.098310 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:07.098359 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 23:50:07.098444 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 23:50:07.098504 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":9}
I20260928 23:50:07.098667 14142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:07.094751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:34573 (local address 127.6.152.1:38171)
0928 23:50:07.094890 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:07.094895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:07.094911 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:50:07.095175 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:07.095177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:07.095226 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:07.095296 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:07.095687 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:07.096268 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:07.097607 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:07.098008 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:07.098153 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:07.098215 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:07.098473 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 23:50:07.098560 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 23:50:07.098664 14120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:07.094663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:46775 (local address 127.6.152.3:38363)
0928 23:50:07.095095 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:07.095110 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:07.095185 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:07.095437 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:07.095439 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:07.095455 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:07.095675 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:07.095682 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:07.096379 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:07.096383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:07.097391 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:07.097400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:07.098141 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:07.098145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:07.098253 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:07.098282 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 23:50:07.098343 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 23:50:07.098451 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":337}
I20260928 23:50:07.099313 13792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1266a5341424db092b36df8cfcc76cc" candidate_uuid: "24026d9eea29427bb69abcf470ecf1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af5a3b2b50fa4554a12490c58ccbb825" is_pre_election: true
I20260928 23:50:07.099310 13929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1266a5341424db092b36df8cfcc76cc" candidate_uuid: "24026d9eea29427bb69abcf470ecf1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b865839cc84946afb5399f87beee52" is_pre_election: true
I20260928 23:50:07.099603 13929 raft_consensus.cc:2454] T f1266a5341424db092b36df8cfcc76cc P 90b865839cc84946afb5399f87beee52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24026d9eea29427bb69abcf470ecf1da in term 0.
I20260928 23:50:07.099617 13792 raft_consensus.cc:2454] T f1266a5341424db092b36df8cfcc76cc P af5a3b2b50fa4554a12490c58ccbb825 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24026d9eea29427bb69abcf470ecf1da in term 0.
I20260928 23:50:07.100054 14001 leader_election.cc:302] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24026d9eea29427bb69abcf470ecf1da, af5a3b2b50fa4554a12490c58ccbb825; no voters:
I20260928 23:50:07.100301 14153 raft_consensus.cc:2781] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:50:07.100386 14153 raft_consensus.cc:482] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:50:07.100446 14153 raft_consensus.cc:3037] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:07.101506 14153 raft_consensus.cc:504] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5a3b2b50fa4554a12490c58ccbb825" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 38171 } } peers { permanent_uuid: "24026d9eea29427bb69abcf470ecf1da" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33437 } } peers { permanent_uuid: "90b865839cc84946afb5399f87beee52" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46775 } }
I20260928 23:50:07.101886 14153 leader_election.cc:288] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da [CANDIDATE]: Term 1 election: Requested vote from peers af5a3b2b50fa4554a12490c58ccbb825 (127.6.152.1:38171), 90b865839cc84946afb5399f87beee52 (127.6.152.2:46775)
I20260928 23:50:07.102208 13792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1266a5341424db092b36df8cfcc76cc" candidate_uuid: "24026d9eea29427bb69abcf470ecf1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af5a3b2b50fa4554a12490c58ccbb825"
I20260928 23:50:07.102324 13792 raft_consensus.cc:3037] T f1266a5341424db092b36df8cfcc76cc P af5a3b2b50fa4554a12490c58ccbb825 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:07.102341 13929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1266a5341424db092b36df8cfcc76cc" candidate_uuid: "24026d9eea29427bb69abcf470ecf1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b865839cc84946afb5399f87beee52"
I20260928 23:50:07.102483 13929 raft_consensus.cc:3037] T f1266a5341424db092b36df8cfcc76cc P 90b865839cc84946afb5399f87beee52 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:07.103439 13792 raft_consensus.cc:2454] T f1266a5341424db092b36df8cfcc76cc P af5a3b2b50fa4554a12490c58ccbb825 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24026d9eea29427bb69abcf470ecf1da in term 1.
I20260928 23:50:07.103729 14001 leader_election.cc:302] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24026d9eea29427bb69abcf470ecf1da, af5a3b2b50fa4554a12490c58ccbb825; no voters:
I20260928 23:50:07.103813 13929 raft_consensus.cc:2454] T f1266a5341424db092b36df8cfcc76cc P 90b865839cc84946afb5399f87beee52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24026d9eea29427bb69abcf470ecf1da in term 1.
I20260928 23:50:07.103889 14153 raft_consensus.cc:2781] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:50:07.104111 14153 raft_consensus.cc:686] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da [term 1 LEADER]: Becoming Leader. State: Replica: 24026d9eea29427bb69abcf470ecf1da, State: Running, Role: LEADER
I20260928 23:50:07.104490 14153 consensus_queue.cc:237] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5a3b2b50fa4554a12490c58ccbb825" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 38171 } } peers { permanent_uuid: "24026d9eea29427bb69abcf470ecf1da" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33437 } } peers { permanent_uuid: "90b865839cc84946afb5399f87beee52" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46775 } }
I20260928 23:50:07.108404 12862 catalog_manager.cc:5495] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da reported cstate change: term changed from 0 to 1, leader changed from <none> to 24026d9eea29427bb69abcf470ecf1da (127.6.152.3). New cstate: current_term: 1 leader_uuid: "24026d9eea29427bb69abcf470ecf1da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5a3b2b50fa4554a12490c58ccbb825" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24026d9eea29427bb69abcf470ecf1da" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90b865839cc84946afb5399f87beee52" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46775 } health_report { overall_health: UNKNOWN } } }
W20260928 23:50:07.133697 13979 tablet.cc:2369] T f1266a5341424db092b36df8cfcc76cc P 90b865839cc84946afb5399f87beee52: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:50:07.172931 14144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:07.169016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52430 (local address 127.6.152.3:33437)
0928 23:50:07.169151 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:07.169156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:07.169170 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:50:07.169295 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:07.169298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:07.169364 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:07.169480 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:50:07.169707 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:07.170237 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:07.171078 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:07.171306 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:07.171411 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:07.171764 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:50:07.172383 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:50:07.172542 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:07.172604 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 23:50:07.172688 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:50:07.178921 13792 raft_consensus.cc:1261] T f1266a5341424db092b36df8cfcc76cc P af5a3b2b50fa4554a12490c58ccbb825 [term 1 FOLLOWER]: Refusing update from remote peer 24026d9eea29427bb69abcf470ecf1da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:50:07.178927 13929 raft_consensus.cc:1261] T f1266a5341424db092b36df8cfcc76cc P 90b865839cc84946afb5399f87beee52 [term 1 FOLLOWER]: Refusing update from remote peer 24026d9eea29427bb69abcf470ecf1da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:50:07.179631 14153 consensus_queue.cc:1035] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da [LEADER]: Connected to new peer: Peer: permanent_uuid: "af5a3b2b50fa4554a12490c58ccbb825" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:50:07.179829 14156 consensus_queue.cc:1035] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b865839cc84946afb5399f87beee52" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:50:07.192935 14160 mvcc.cc:205] Tried to move back new op lower bound from 7334459011797962752 to 7334459011503128576. Current Snapshot: MvccSnapshot[applied={T|T < 7334459011797962752}]
I20260928 23:50:07.193768 14161 mvcc.cc:205] Tried to move back new op lower bound from 7334459011797962752 to 7334459011503128576. Current Snapshot: MvccSnapshot[applied={T|T < 7334459011797962752 or (T in {7334459011797962752})}]
I20260928 23:50:07.198341 12862 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:50:07.198501 12862 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:50:07.202549 12862 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b20d904b67749e9a3276bfdd467c7be: Sending DeleteTablet for 3 replicas of tablet f1266a5341424db092b36df8cfcc76cc
I20260928 23:50:07.203168 13772 tablet_service.cc:1514] Processing DeleteTablet for tablet f1266a5341424db092b36df8cfcc76cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:50:07 UTC) from {username='oryx', principal='oryx/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:50564
I20260928 23:50:07.203671 14168 tablet_replica.cc:330] T f1266a5341424db092b36df8cfcc76cc P af5a3b2b50fa4554a12490c58ccbb825: stopping tablet replica
I20260928 23:50:07.204083 14048 tablet_service.cc:1514] Processing DeleteTablet for tablet f1266a5341424db092b36df8cfcc76cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:50:07 UTC) from {username='oryx', principal='oryx/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:52414
I20260928 23:50:07.204239 14168 raft_consensus.cc:2229] T f1266a5341424db092b36df8cfcc76cc P af5a3b2b50fa4554a12490c58ccbb825 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:50:07.204561 14168 raft_consensus.cc:2258] T f1266a5341424db092b36df8cfcc76cc P af5a3b2b50fa4554a12490c58ccbb825 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:50:07.204519 13909 tablet_service.cc:1514] Processing DeleteTablet for tablet f1266a5341424db092b36df8cfcc76cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:50:07 UTC) from {username='oryx', principal='oryx/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:58822
I20260928 23:50:07.205403 14169 tablet_replica.cc:330] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da: stopping tablet replica
I20260928 23:50:07.205670 14169 raft_consensus.cc:2229] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:50:07.205932 14168 ts_tablet_manager.cc:1905] T f1266a5341424db092b36df8cfcc76cc P af5a3b2b50fa4554a12490c58ccbb825: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:50:07.205974 14169 raft_consensus.cc:2258] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:50:07.206259 14170 tablet_replica.cc:330] T f1266a5341424db092b36df8cfcc76cc P 90b865839cc84946afb5399f87beee52: stopping tablet replica
I20260928 23:50:07.206523 14170 raft_consensus.cc:2229] T f1266a5341424db092b36df8cfcc76cc P 90b865839cc84946afb5399f87beee52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:50:07.206776 14170 raft_consensus.cc:2258] T f1266a5341424db092b36df8cfcc76cc P 90b865839cc84946afb5399f87beee52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:50:07.208034 14170 ts_tablet_manager.cc:1905] T f1266a5341424db092b36df8cfcc76cc P 90b865839cc84946afb5399f87beee52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:50:07.208252 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 12830
I20260928 23:50:07.209908 14168 ts_tablet_manager.cc:1918] T f1266a5341424db092b36df8cfcc76cc P af5a3b2b50fa4554a12490c58ccbb825: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:50:07.210012 14168 log.cc:1196] T f1266a5341424db092b36df8cfcc76cc P af5a3b2b50fa4554a12490c58ccbb825: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-0/wal/wals/f1266a5341424db092b36df8cfcc76cc
I20260928 23:50:07.210335 14168 ts_tablet_manager.cc:1939] T f1266a5341424db092b36df8cfcc76cc P af5a3b2b50fa4554a12490c58ccbb825: Deleting consensus metadata
I20260928 23:50:07.211181 14170 ts_tablet_manager.cc:1918] T f1266a5341424db092b36df8cfcc76cc P 90b865839cc84946afb5399f87beee52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:50:07.211280 14170 log.cc:1196] T f1266a5341424db092b36df8cfcc76cc P 90b865839cc84946afb5399f87beee52: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/wal/wals/f1266a5341424db092b36df8cfcc76cc
I20260928 23:50:07.211582 14170 ts_tablet_manager.cc:1939] T f1266a5341424db092b36df8cfcc76cc P 90b865839cc84946afb5399f87beee52: Deleting consensus metadata
I20260928 23:50:07.213860 14169 ts_tablet_manager.cc:1905] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 23:50:07.216193 13864 connection.cc:535] server connection from 127.0.0.1:58822 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 23:50:07.216517 13726 connection.cc:535] server connection from 127.0.0.1:50564 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 23:50:07.216943 14169 ts_tablet_manager.cc:1918] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:50:07.217039 14169 log.cc:1196] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/f1266a5341424db092b36df8cfcc76cc
I20260928 23:50:07.217372 14169 ts_tablet_manager.cc:1939] T f1266a5341424db092b36df8cfcc76cc P 24026d9eea29427bb69abcf470ecf1da: Deleting consensus metadata
W20260928 23:50:07.218428 14000 connection.cc:441] server connection from 127.0.0.1:52414 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52414 (request call id 1) could send its response
I20260928 23:50:07.227636 14126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:07.206808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52686 (local address 127.6.152.61:46027)
0928 23:50:07.206916 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:07.206931 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:07.216778 (+ 9847us) server_negotiation.cc:407] Connection header received
0928 23:50:07.217023 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:07.217026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:07.217077 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:07.217127 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:07.218174 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:07.219002 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:07.220278 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:07.220620 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:07.220733 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:07.223093 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:07.223097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:07.223100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:07.223124 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:07.224936 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:07.225394 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:07.225397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:07.225399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:07.225503 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:07.226154 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:07.226158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:07.226160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:07.226399 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:07.226540 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:07.227297 (+ 757us) server_negotiation.cc:299] Negotiation successful
0928 23:50:07.227450 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:50:07.231528 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:07.221734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52700 (local address 127.6.152.61:46027)
0928 23:50:07.221839 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:07.221843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:07.221856 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:50:07.221922 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:07.221925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:07.221968 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:07.222024 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:07.222723 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:07.223364 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:07.224655 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:07.225052 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:07.225123 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:07.226802 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:07.226805 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:07.226808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:07.226829 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:07.229809 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:07.230251 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:07.230254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:07.230256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:07.230338 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:07.230637 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:07.230640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:07.230642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:07.230831 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:07.230930 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:07.231241 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 23:50:07.231361 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:50:07.232060 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:07.221361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39226 (local address 127.6.152.60:34469)
0928 23:50:07.221484 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:07.221489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:07.223365 (+ 1876us) server_negotiation.cc:407] Connection header received
0928 23:50:07.223547 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:07.223550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:07.223601 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:07.223651 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:07.224490 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:07.225281 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:07.226477 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:07.226818 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:07.226885 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:07.228401 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:07.228405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:07.228408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:07.228437 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:07.230276 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:07.230748 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:07.230752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:07.230754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:07.230835 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:07.231132 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:07.231136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:07.231138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:07.231332 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:07.231442 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:07.231799 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 23:50:07.231912 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:50:07.237377 14129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:07.206522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39210 (local address 127.6.152.60:34469)
0928 23:50:07.206626 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:07.206630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:07.227488 (+ 20858us) server_negotiation.cc:407] Connection header received
0928 23:50:07.227531 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:07.227534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:07.227582 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:07.227652 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:07.229493 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:07.230285 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:07.231344 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:07.231698 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:07.231796 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:07.233956 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:07.233959 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:07.233961 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:07.233981 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:07.235344 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:07.236018 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:07.236023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:07.236025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:07.236146 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:07.236442 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:07.236446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:07.236447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:07.236659 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:07.236776 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:07.237086 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 23:50:07.237257 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 23:50:08.192889 14178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.192352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.6.152.2:39459)
0928 23:50:08.192608 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:08.192714 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":74,"threads_started":1}
W20260928 23:50:08.193558 13977 heartbeater.cc:644] Failed to heartbeat to 127.6.152.62:36193 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
I20260928 23:50:08.195681 14179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.195216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.6.152.3:40927)
0928 23:50:08.195506 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:08.195589 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":88,"threads_started":1}
W20260928 23:50:08.195974 14115 heartbeater.cc:644] Failed to heartbeat to 127.6.152.62:36193 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
I20260928 23:50:08.196350 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.195976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.6.152.1:48317)
0928 23:50:08.196213 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:08.196277 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":61,"threads_started":1}
W20260928 23:50:08.196614 13839 heartbeater.cc:644] Failed to heartbeat to 127.6.152.62:36193 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
I20260928 23:50:08.702834 14181 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b20d904b67749e9a3276bfdd467c7be)
I20260928 23:50:08.703094 14181 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: false } }
I20260928 23:50:08.703966 14182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b20d904b67749e9a3276bfdd467c7be)
I20260928 23:50:08.704186 14182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: false } }
I20260928 23:50:08.704288 14181 leader_election.cc:288] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b20d904b67749e9a3276bfdd467c7be (127.6.152.62:36193), 8c611b33841b41618e43a93d54e67fd7 (127.6.152.61:46027)
I20260928 23:50:08.704972 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.704424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.0.0.1:53564)
0928 23:50:08.704748 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:08.704848 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":77,"threads_started":1}
W20260928 23:50:08.705205 13644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
I20260928 23:50:08.705246 14182 leader_election.cc:288] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b20d904b67749e9a3276bfdd467c7be (127.6.152.62:36193), ec71e3446d104e96b5462e085b9e757b (127.6.152.60:34469)
I20260928 23:50:08.706028 14188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.705302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.0.0.1:53578)
0928 23:50:08.705864 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:08.705935 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":60,"threads_started":1}
I20260928 23:50:08.706203 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.705957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.0.0.1:53590)
0928 23:50:08.706096 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:08.706142 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260928 23:50:08.706179 13458 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
W20260928 23:50:08.706293 13644 leader_election.cc:334] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b20d904b67749e9a3276bfdd467c7be (127.6.152.62:36193): Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
I20260928 23:50:08.707115 14188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.706812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.0.0.1:53606)
0928 23:50:08.706934 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:08.706988 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260928 23:50:08.707237 13458 leader_election.cc:334] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b20d904b67749e9a3276bfdd467c7be (127.6.152.62:36193): Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
I20260928 23:50:08.709291 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.704608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52710 (local address 127.6.152.61:46027)
0928 23:50:08.704971 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:08.704975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:08.704995 (+ 20us) server_negotiation.cc:407] Connection header received
0928 23:50:08.705101 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:08.705104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:08.705160 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:08.705248 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:08.705692 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:08.706747 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:08.708229 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:08.708725 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:08.708873 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:08.708935 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:08.709130 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 23:50:08.709203 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":285,"thread_start_us":52,"threads_started":1}
I20260928 23:50:08.709344 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.704534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:46027 (local address 127.0.0.1:52710)
0928 23:50:08.704830 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:08.704845 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:08.704950 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:08.705366 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:08.705371 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:08.705396 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:08.705520 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:08.705526 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:08.706900 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:08.706902 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:08.708119 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:08.708126 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:08.708877 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:08.708881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:08.708975 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:08.709000 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 23:50:08.709080 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 23:50:08.709217 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":57,"thread_start_us":70,"threads_started":1}
I20260928 23:50:08.709830 13484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec71e3446d104e96b5462e085b9e757b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8c611b33841b41618e43a93d54e67fd7" is_pre_election: true
I20260928 23:50:08.710054 13484 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec71e3446d104e96b5462e085b9e757b in term 1.
I20260928 23:50:08.710270 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.705895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39236 (local address 127.6.152.60:34469)
0928 23:50:08.706541 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:08.706551 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:08.706563 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:50:08.706606 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:08.706609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:08.706645 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:08.706756 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:08.707132 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:08.707878 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:08.709002 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:08.709634 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:08.709755 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:08.709807 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:08.710088 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 23:50:08.710163 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":90,"threads_started":1}
I20260928 23:50:08.710274 14186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.705675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:34469 (local address 127.0.0.1:39236)
0928 23:50:08.706134 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:08.706150 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:08.706232 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:08.706792 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:08.706797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:08.706814 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:08.706908 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:08.706914 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:08.708000 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:08.708003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:08.708903 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:08.708919 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:08.709764 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:08.709768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:08.709870 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:08.709889 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 23:50:08.710070 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 23:50:08.710146 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":2,"thread_start_us":61,"threads_started":1}
I20260928 23:50:08.710392 13643 leader_election.cc:302] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c611b33841b41618e43a93d54e67fd7, ec71e3446d104e96b5462e085b9e757b; no voters: 8b20d904b67749e9a3276bfdd467c7be
I20260928 23:50:08.710632 14181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 23:50:08.710726 14181 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8b20d904b67749e9a3276bfdd467c7be)
I20260928 23:50:08.710716 13663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c611b33841b41618e43a93d54e67fd7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ec71e3446d104e96b5462e085b9e757b" is_pre_election: true
I20260928 23:50:08.710893 14181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:50:08.711927 14181 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: false } }
I20260928 23:50:08.712175 13663 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c611b33841b41618e43a93d54e67fd7 in current term 2: Already voted for candidate ec71e3446d104e96b5462e085b9e757b in this term.
I20260928 23:50:08.712329 14181 leader_election.cc:288] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [CANDIDATE]: Term 2 election: Requested vote from peers 8b20d904b67749e9a3276bfdd467c7be (127.6.152.62:36193), 8c611b33841b41618e43a93d54e67fd7 (127.6.152.61:46027)
I20260928 23:50:08.712648 13484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec71e3446d104e96b5462e085b9e757b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8c611b33841b41618e43a93d54e67fd7"
I20260928 23:50:08.712754 13484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:50:08.713223 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.712994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.0.0.1:53618)
0928 23:50:08.713117 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:08.713168 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 23:50:08.713255 13460 leader_election.cc:302] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c611b33841b41618e43a93d54e67fd7; no voters: 8b20d904b67749e9a3276bfdd467c7be, ec71e3446d104e96b5462e085b9e757b
I20260928 23:50:08.714069 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.713851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.0.0.1:53626)
0928 23:50:08.713981 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:08.714024 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 23:50:08.714057 13484 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec71e3446d104e96b5462e085b9e757b in term 2.
W20260928 23:50:08.714174 13644 leader_election.cc:334] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8b20d904b67749e9a3276bfdd467c7be (127.6.152.62:36193): Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
I20260928 23:50:08.714185 14182 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:50:08.714407 13643 leader_election.cc:302] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c611b33841b41618e43a93d54e67fd7, ec71e3446d104e96b5462e085b9e757b; no voters: 8b20d904b67749e9a3276bfdd467c7be
I20260928 23:50:08.714577 14181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [term 2 FOLLOWER]: Leader election won for term 2
I20260928 23:50:08.714735 14181 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [term 2 LEADER]: Becoming Leader. State: Replica: ec71e3446d104e96b5462e085b9e757b, State: Running, Role: LEADER
I20260928 23:50:08.714968 14181 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: false } }
I20260928 23:50:08.716190 14191 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec71e3446d104e96b5462e085b9e757b. Latest consensus state: current_term: 2 leader_uuid: "ec71e3446d104e96b5462e085b9e757b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: false } } }
I20260928 23:50:08.716305 14191 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:08.716574 14193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:50:08.718755 14193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:50:08.719170 14193 catalog_manager.cc:1259] Loaded cluster ID: cef39238cd1d47c9b5fe97b10e1b2ace
I20260928 23:50:08.719236 14193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:50:08.719842 14193 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:50:08.720263 14193 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b: Loaded TSK: 0
I20260928 23:50:08.721016 14193 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:50:08.752094 13659 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:50:08.755224 13659 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:50:08.758141 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.757908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.0.0.1:53644)
0928 23:50:08.758023 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:08.758077 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 23:50:08.758258 13484 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [term 2 FOLLOWER]: Refusing update from remote peer ec71e3446d104e96b5462e085b9e757b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 23:50:08.758837 14181 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 23:50:08.758917 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.758653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.0.0.1:53660)
0928 23:50:08.758749 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:08.758815 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 23:50:08.759042 13644 consensus_peers.cc:487] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b -> Peer 8b20d904b67749e9a3276bfdd467c7be (127.6.152.62:36193): Couldn't send request to peer 8b20d904b67749e9a3276bfdd467c7be. Status: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 23:50:08.760263 14182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec71e3446d104e96b5462e085b9e757b. Latest consensus state: current_term: 2 leader_uuid: "ec71e3446d104e96b5462e085b9e757b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: false } } }
I20260928 23:50:08.760401 14182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:50:08.761428 14181 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ec71e3446d104e96b5462e085b9e757b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: false } } }
I20260928 23:50:08.761541 14181 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:08.761718 14182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ec71e3446d104e96b5462e085b9e757b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8b20d904b67749e9a3276bfdd467c7be" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36193 } } peers { permanent_uuid: "8c611b33841b41618e43a93d54e67fd7" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 46027 } attrs { promote: false } } peers { permanent_uuid: "ec71e3446d104e96b5462e085b9e757b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 34469 } attrs { promote: false } } }
I20260928 23:50:08.761830 14182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c611b33841b41618e43a93d54e67fd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:50:08.772275 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.767773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:46775 (local address 127.0.0.1:58828)
0928 23:50:08.768268 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:08.768280 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:08.768394 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:08.768829 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:08.768833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:08.768848 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:08.768963 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:08.768969 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:08.770059 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:08.770062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:08.771079 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:08.771088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:08.771925 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:08.771929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:08.771988 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:08.772004 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 23:50:08.772080 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 23:50:08.772148 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":57}
I20260928 23:50:08.772290 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.767875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58828 (local address 127.6.152.2:46775)
0928 23:50:08.768126 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:08.768132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:08.768338 (+ 206us) server_negotiation.cc:407] Connection header received
0928 23:50:08.768532 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:08.768538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:08.768598 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:08.768658 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:08.769087 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:08.769920 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:08.771209 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:08.771740 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:08.771804 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:08.771854 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:08.772108 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 23:50:08.772177 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":74,"threads_started":1}
I20260928 23:50:08.772584 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.768042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:38171 (local address 127.0.0.1:50574)
0928 23:50:08.768666 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:08.768679 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:08.768829 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:08.769078 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:08.769081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:08.769096 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:08.769177 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:08.769183 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:08.770093 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:08.770100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:08.771346 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:08.771357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:08.772252 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:08.772256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:08.772349 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:08.772374 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 23:50:08.772431 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 23:50:08.772492 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":31,"thread_start_us":122,"threads_started":1}
I20260928 23:50:08.773118 13909 tablet_service.cc:1467] Processing CreateTablet for tablet d12a0b9d82d94e4ab9c2b297a49af8bf (DEFAULT_TABLE table=test-table [id=1db250a8025b4ffe976c3f681ed660f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:50:08.773195 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.767530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:33437 (local address 127.0.0.1:52446)
0928 23:50:08.768176 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:08.768191 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:08.768278 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:08.769278 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:08.769283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:08.769301 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:08.769374 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:08.769377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:08.770276 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:08.770279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:08.771385 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:08.771395 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:08.772252 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:08.772256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:08.772912 (+ 656us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:08.772938 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 23:50:08.773024 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 23:50:08.773090 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":535}
I20260928 23:50:08.773245 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.767628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52446 (local address 127.6.152.3:33437)
0928 23:50:08.769056 (+ 1428us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:08.769062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:08.769077 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:50:08.769135 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:08.769137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:08.769183 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:08.769265 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:08.769473 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:08.770180 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:08.771512 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:08.772125 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:08.772299 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:08.772360 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:08.773041 (+ 681us) server_negotiation.cc:299] Negotiation successful
0928 23:50:08.773127 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1341,"thread_start_us":72,"threads_started":1}
I20260928 23:50:08.773367 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.768449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50574 (local address 127.6.152.1:38171)
0928 23:50:08.768692 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:08.768696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:08.768712 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:50:08.768934 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:08.768937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:08.768986 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:08.769073 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:08.769489 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:08.769986 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:08.771503 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:08.772115 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:08.772934 (+ 819us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:08.772997 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:08.773033 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 23:50:08.773101 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":79,"threads_started":1}
I20260928 23:50:08.773427 13909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d12a0b9d82d94e4ab9c2b297a49af8bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:08.773722 14048 tablet_service.cc:1467] Processing CreateTablet for tablet d12a0b9d82d94e4ab9c2b297a49af8bf (DEFAULT_TABLE table=test-table [id=1db250a8025b4ffe976c3f681ed660f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:50:08.773937 14048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d12a0b9d82d94e4ab9c2b297a49af8bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:08.775497 14203 tablet_bootstrap.cc:492] T d12a0b9d82d94e4ab9c2b297a49af8bf P 90b865839cc84946afb5399f87beee52: Bootstrap starting.
I20260928 23:50:08.776047 14204 tablet_bootstrap.cc:492] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da: Bootstrap starting.
I20260928 23:50:08.776605 14203 tablet_bootstrap.cc:654] T d12a0b9d82d94e4ab9c2b297a49af8bf P 90b865839cc84946afb5399f87beee52: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:08.776561 13772 tablet_service.cc:1467] Processing CreateTablet for tablet d12a0b9d82d94e4ab9c2b297a49af8bf (DEFAULT_TABLE table=test-table [id=1db250a8025b4ffe976c3f681ed660f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:50:08.776896 13772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d12a0b9d82d94e4ab9c2b297a49af8bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:08.776974 14204 tablet_bootstrap.cc:654] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:08.777806 14203 tablet_bootstrap.cc:492] T d12a0b9d82d94e4ab9c2b297a49af8bf P 90b865839cc84946afb5399f87beee52: No bootstrap required, opened a new log
I20260928 23:50:08.777954 14203 ts_tablet_manager.cc:1397] T d12a0b9d82d94e4ab9c2b297a49af8bf P 90b865839cc84946afb5399f87beee52: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:50:08.778332 14203 raft_consensus.cc:348] T d12a0b9d82d94e4ab9c2b297a49af8bf P 90b865839cc84946afb5399f87beee52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24026d9eea29427bb69abcf470ecf1da" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33437 } } peers { permanent_uuid: "90b865839cc84946afb5399f87beee52" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46775 } } peers { permanent_uuid: "af5a3b2b50fa4554a12490c58ccbb825" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 38171 } }
I20260928 23:50:08.778450 14203 raft_consensus.cc:374] T d12a0b9d82d94e4ab9c2b297a49af8bf P 90b865839cc84946afb5399f87beee52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:08.778492 14203 raft_consensus.cc:729] T d12a0b9d82d94e4ab9c2b297a49af8bf P 90b865839cc84946afb5399f87beee52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b865839cc84946afb5399f87beee52, State: Initialized, Role: FOLLOWER
I20260928 23:50:08.778627 14203 consensus_queue.cc:260] T d12a0b9d82d94e4ab9c2b297a49af8bf P 90b865839cc84946afb5399f87beee52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24026d9eea29427bb69abcf470ecf1da" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33437 } } peers { permanent_uuid: "90b865839cc84946afb5399f87beee52" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46775 } } peers { permanent_uuid: "af5a3b2b50fa4554a12490c58ccbb825" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 38171 } }
I20260928 23:50:08.778983 14203 ts_tablet_manager.cc:1428] T d12a0b9d82d94e4ab9c2b297a49af8bf P 90b865839cc84946afb5399f87beee52: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:50:08.779810 14204 tablet_bootstrap.cc:492] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da: No bootstrap required, opened a new log
I20260928 23:50:08.779922 14204 ts_tablet_manager.cc:1397] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 23:50:08.780354 14204 raft_consensus.cc:348] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24026d9eea29427bb69abcf470ecf1da" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33437 } } peers { permanent_uuid: "90b865839cc84946afb5399f87beee52" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46775 } } peers { permanent_uuid: "af5a3b2b50fa4554a12490c58ccbb825" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 38171 } }
I20260928 23:50:08.780478 14204 raft_consensus.cc:374] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:08.780520 14204 raft_consensus.cc:729] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24026d9eea29427bb69abcf470ecf1da, State: Initialized, Role: FOLLOWER
I20260928 23:50:08.780674 14204 consensus_queue.cc:260] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24026d9eea29427bb69abcf470ecf1da" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33437 } } peers { permanent_uuid: "90b865839cc84946afb5399f87beee52" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46775 } } peers { permanent_uuid: "af5a3b2b50fa4554a12490c58ccbb825" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 38171 } }
I20260928 23:50:08.780862 14208 tablet_bootstrap.cc:492] T d12a0b9d82d94e4ab9c2b297a49af8bf P af5a3b2b50fa4554a12490c58ccbb825: Bootstrap starting.
I20260928 23:50:08.781029 14204 ts_tablet_manager.cc:1428] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:50:08.781533 13976 heartbeater.cc:503] Master 127.6.152.60:34469 was elected leader, sending a full tablet report...
I20260928 23:50:08.781886 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.779731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.6.152.2:56733)
0928 23:50:08.781704 (+ 1973us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:08.781764 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1906,"thread_start_us":104,"threads_started":1}
I20260928 23:50:08.782485 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.782108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.6.152.3:53587)
0928 23:50:08.782330 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:08.782389 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":95,"threads_started":1}
I20260928 23:50:08.782509 14114 heartbeater.cc:503] Master 127.6.152.60:34469 was elected leader, sending a full tablet report...
I20260928 23:50:08.783218 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.783020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.6.152.3:51329)
0928 23:50:08.783107 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:08.783156 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 23:50:08.783833 14208 tablet_bootstrap.cc:654] T d12a0b9d82d94e4ab9c2b297a49af8bf P af5a3b2b50fa4554a12490c58ccbb825: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:08.785099 14208 tablet_bootstrap.cc:492] T d12a0b9d82d94e4ab9c2b297a49af8bf P af5a3b2b50fa4554a12490c58ccbb825: No bootstrap required, opened a new log
I20260928 23:50:08.785207 14208 ts_tablet_manager.cc:1397] T d12a0b9d82d94e4ab9c2b297a49af8bf P af5a3b2b50fa4554a12490c58ccbb825: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 23:50:08.785651 14208 raft_consensus.cc:348] T d12a0b9d82d94e4ab9c2b297a49af8bf P af5a3b2b50fa4554a12490c58ccbb825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24026d9eea29427bb69abcf470ecf1da" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33437 } } peers { permanent_uuid: "90b865839cc84946afb5399f87beee52" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46775 } } peers { permanent_uuid: "af5a3b2b50fa4554a12490c58ccbb825" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 38171 } }
I20260928 23:50:08.785761 14208 raft_consensus.cc:374] T d12a0b9d82d94e4ab9c2b297a49af8bf P af5a3b2b50fa4554a12490c58ccbb825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:08.785805 14208 raft_consensus.cc:729] T d12a0b9d82d94e4ab9c2b297a49af8bf P af5a3b2b50fa4554a12490c58ccbb825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af5a3b2b50fa4554a12490c58ccbb825, State: Initialized, Role: FOLLOWER
I20260928 23:50:08.785979 14208 consensus_queue.cc:260] T d12a0b9d82d94e4ab9c2b297a49af8bf P af5a3b2b50fa4554a12490c58ccbb825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24026d9eea29427bb69abcf470ecf1da" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33437 } } peers { permanent_uuid: "90b865839cc84946afb5399f87beee52" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46775 } } peers { permanent_uuid: "af5a3b2b50fa4554a12490c58ccbb825" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 38171 } }
I20260928 23:50:08.786337 14208 ts_tablet_manager.cc:1428] T d12a0b9d82d94e4ab9c2b297a49af8bf P af5a3b2b50fa4554a12490c58ccbb825: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 23:50:08.787343 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.786649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.6.152.1:41447)
0928 23:50:08.787017 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:08.787077 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":90,"threads_started":1}
I20260928 23:50:08.787678 13838 heartbeater.cc:503] Master 127.6.152.60:34469 was elected leader, sending a full tablet report...
I20260928 23:50:08.788290 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.787757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.6.152.1:54515)
0928 23:50:08.788180 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:08.788230 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352}
I20260928 23:50:08.973510 14210 raft_consensus.cc:482] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:50:08.973666 14210 raft_consensus.cc:504] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24026d9eea29427bb69abcf470ecf1da" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33437 } } peers { permanent_uuid: "90b865839cc84946afb5399f87beee52" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46775 } } peers { permanent_uuid: "af5a3b2b50fa4554a12490c58ccbb825" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 38171 } }
I20260928 23:50:08.974153 14210 leader_election.cc:288] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90b865839cc84946afb5399f87beee52 (127.6.152.2:46775), af5a3b2b50fa4554a12490c58ccbb825 (127.6.152.1:38171)
I20260928 23:50:08.974634 13929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d12a0b9d82d94e4ab9c2b297a49af8bf" candidate_uuid: "24026d9eea29427bb69abcf470ecf1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b865839cc84946afb5399f87beee52" is_pre_election: true
I20260928 23:50:08.974786 13929 raft_consensus.cc:2454] T d12a0b9d82d94e4ab9c2b297a49af8bf P 90b865839cc84946afb5399f87beee52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24026d9eea29427bb69abcf470ecf1da in term 0.
I20260928 23:50:08.974764 13792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d12a0b9d82d94e4ab9c2b297a49af8bf" candidate_uuid: "24026d9eea29427bb69abcf470ecf1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af5a3b2b50fa4554a12490c58ccbb825" is_pre_election: true
I20260928 23:50:08.974978 13792 raft_consensus.cc:2454] T d12a0b9d82d94e4ab9c2b297a49af8bf P af5a3b2b50fa4554a12490c58ccbb825 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24026d9eea29427bb69abcf470ecf1da in term 0.
I20260928 23:50:08.975171 14000 leader_election.cc:302] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24026d9eea29427bb69abcf470ecf1da, 90b865839cc84946afb5399f87beee52; no voters:
I20260928 23:50:08.975345 14210 raft_consensus.cc:2781] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:50:08.975433 14210 raft_consensus.cc:482] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:50:08.975487 14210 raft_consensus.cc:3037] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:08.976511 14210 raft_consensus.cc:504] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24026d9eea29427bb69abcf470ecf1da" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33437 } } peers { permanent_uuid: "90b865839cc84946afb5399f87beee52" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46775 } } peers { permanent_uuid: "af5a3b2b50fa4554a12490c58ccbb825" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 38171 } }
I20260928 23:50:08.976910 14210 leader_election.cc:288] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da [CANDIDATE]: Term 1 election: Requested vote from peers 90b865839cc84946afb5399f87beee52 (127.6.152.2:46775), af5a3b2b50fa4554a12490c58ccbb825 (127.6.152.1:38171)
I20260928 23:50:08.977224 13929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d12a0b9d82d94e4ab9c2b297a49af8bf" candidate_uuid: "24026d9eea29427bb69abcf470ecf1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b865839cc84946afb5399f87beee52"
I20260928 23:50:08.977305 13792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d12a0b9d82d94e4ab9c2b297a49af8bf" candidate_uuid: "24026d9eea29427bb69abcf470ecf1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af5a3b2b50fa4554a12490c58ccbb825"
I20260928 23:50:08.977368 13929 raft_consensus.cc:3037] T d12a0b9d82d94e4ab9c2b297a49af8bf P 90b865839cc84946afb5399f87beee52 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:08.977455 13792 raft_consensus.cc:3037] T d12a0b9d82d94e4ab9c2b297a49af8bf P af5a3b2b50fa4554a12490c58ccbb825 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:08.978281 13929 raft_consensus.cc:2454] T d12a0b9d82d94e4ab9c2b297a49af8bf P 90b865839cc84946afb5399f87beee52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24026d9eea29427bb69abcf470ecf1da in term 1.
I20260928 23:50:08.978386 13792 raft_consensus.cc:2454] T d12a0b9d82d94e4ab9c2b297a49af8bf P af5a3b2b50fa4554a12490c58ccbb825 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24026d9eea29427bb69abcf470ecf1da in term 1.
I20260928 23:50:08.978564 14000 leader_election.cc:302] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24026d9eea29427bb69abcf470ecf1da, 90b865839cc84946afb5399f87beee52; no voters:
I20260928 23:50:08.978734 14210 raft_consensus.cc:2781] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:50:08.978920 14210 raft_consensus.cc:686] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da [term 1 LEADER]: Becoming Leader. State: Replica: 24026d9eea29427bb69abcf470ecf1da, State: Running, Role: LEADER
I20260928 23:50:08.979095 14210 consensus_queue.cc:237] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24026d9eea29427bb69abcf470ecf1da" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33437 } } peers { permanent_uuid: "90b865839cc84946afb5399f87beee52" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46775 } } peers { permanent_uuid: "af5a3b2b50fa4554a12490c58ccbb825" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 38171 } }
I20260928 23:50:08.979810 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:08.979540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.6.152.3:37875)
0928 23:50:08.979663 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:08.979725 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 23:50:08.980748 13658 catalog_manager.cc:5495] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da reported cstate change: term changed from 0 to 1, leader changed from <none> to 24026d9eea29427bb69abcf470ecf1da (127.6.152.3). New cstate: current_term: 1 leader_uuid: "24026d9eea29427bb69abcf470ecf1da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24026d9eea29427bb69abcf470ecf1da" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 33437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90b865839cc84946afb5399f87beee52" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af5a3b2b50fa4554a12490c58ccbb825" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 38171 } health_report { overall_health: UNKNOWN } } }
I20260928 23:50:09.008975 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:09.005798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52462 (local address 127.6.152.3:33437)
0928 23:50:09.005981 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:09.005986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:09.006002 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:50:09.006059 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:09.006061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:09.006105 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:09.006188 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:50:09.006451 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:09.006970 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:09.007879 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:09.008116 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:09.008228 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:09.008497 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:50:09.008570 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:50:09.008648 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:09.008746 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 23:50:09.008799 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 23:50:09.011233 13792 raft_consensus.cc:1261] T d12a0b9d82d94e4ab9c2b297a49af8bf P af5a3b2b50fa4554a12490c58ccbb825 [term 1 FOLLOWER]: Refusing update from remote peer 24026d9eea29427bb69abcf470ecf1da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:50:09.011221 13929 raft_consensus.cc:1261] T d12a0b9d82d94e4ab9c2b297a49af8bf P 90b865839cc84946afb5399f87beee52 [term 1 FOLLOWER]: Refusing update from remote peer 24026d9eea29427bb69abcf470ecf1da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:50:09.011659 14215 consensus_queue.cc:1035] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da [LEADER]: Connected to new peer: Peer: permanent_uuid: "af5a3b2b50fa4554a12490c58ccbb825" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:50:09.011829 14210 consensus_queue.cc:1035] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b865839cc84946afb5399f87beee52" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:50:09.013746 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:09.013463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.6.152.2:37501)
0928 23:50:09.013610 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:09.013666 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 23:50:09.013885 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:09.013446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.6.152.1:46089)
0928 23:50:09.013753 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:09.013812 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236}
I20260928 23:50:09.015129 14218 mvcc.cc:205] Tried to move back new op lower bound from 7334459019305480192 to 7334459019179741184. Current Snapshot: MvccSnapshot[applied={T|T < 7334459019305480192}]
I20260928 23:50:09.016574 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:09.016307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.6.152.2:60507)
0928 23:50:09.016455 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:09.016505 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 23:50:09.016956 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:09.016637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.6.152.3:48599)
0928 23:50:09.016796 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:09.016881 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 23:50:09.018612 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:09.018406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.6.152.3:39853)
0928 23:50:09.018499 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:09.018548 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 23:50:09.019739 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:09.019464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36193 (local address 127.6.152.1:33445)
0928 23:50:09.019596 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:09.019650 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 23:50:09.024459 13659 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:50:09.024588 13659 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:50:09.028278 13659 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec71e3446d104e96b5462e085b9e757b: Sending DeleteTablet for 3 replicas of tablet d12a0b9d82d94e4ab9c2b297a49af8bf
I20260928 23:50:09.028892 14048 tablet_service.cc:1514] Processing DeleteTablet for tablet d12a0b9d82d94e4ab9c2b297a49af8bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:50:09 UTC) from {username='oryx', principal='oryx/127.6.152.60@KRBTEST.COM'} at 127.0.0.1:52446
I20260928 23:50:09.029062 13909 tablet_service.cc:1514] Processing DeleteTablet for tablet d12a0b9d82d94e4ab9c2b297a49af8bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:50:09 UTC) from {username='oryx', principal='oryx/127.6.152.60@KRBTEST.COM'} at 127.0.0.1:58828
I20260928 23:50:09.029260 14227 tablet_replica.cc:330] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da: stopping tablet replica
I20260928 23:50:09.029392 14227 raft_consensus.cc:2229] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:50:09.029559 13772 tablet_service.cc:1514] Processing DeleteTablet for tablet d12a0b9d82d94e4ab9c2b297a49af8bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:50:09 UTC) from {username='oryx', principal='oryx/127.6.152.60@KRBTEST.COM'} at 127.0.0.1:50574
I20260928 23:50:09.029628 14227 raft_consensus.cc:2258] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:50:09.029632 14228 tablet_replica.cc:330] T d12a0b9d82d94e4ab9c2b297a49af8bf P 90b865839cc84946afb5399f87beee52: stopping tablet replica
I20260928 23:50:09.029758 14228 raft_consensus.cc:2229] T d12a0b9d82d94e4ab9c2b297a49af8bf P 90b865839cc84946afb5399f87beee52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:50:09.029893 14228 raft_consensus.cc:2258] T d12a0b9d82d94e4ab9c2b297a49af8bf P 90b865839cc84946afb5399f87beee52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:50:09.030079 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 13620
I20260928 23:50:09.030275 14229 tablet_replica.cc:330] T d12a0b9d82d94e4ab9c2b297a49af8bf P af5a3b2b50fa4554a12490c58ccbb825: stopping tablet replica
I20260928 23:50:09.030287 14227 ts_tablet_manager.cc:1905] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:50:09.030427 14229 raft_consensus.cc:2229] T d12a0b9d82d94e4ab9c2b297a49af8bf P af5a3b2b50fa4554a12490c58ccbb825 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:50:09.031195 14228 ts_tablet_manager.cc:1905] T d12a0b9d82d94e4ab9c2b297a49af8bf P 90b865839cc84946afb5399f87beee52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:50:09.033299 14228 ts_tablet_manager.cc:1918] T d12a0b9d82d94e4ab9c2b297a49af8bf P 90b865839cc84946afb5399f87beee52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:50:09.033376 14228 log.cc:1196] T d12a0b9d82d94e4ab9c2b297a49af8bf P 90b865839cc84946afb5399f87beee52: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-1/wal/wals/d12a0b9d82d94e4ab9c2b297a49af8bf
I20260928 23:50:09.033595 14227 ts_tablet_manager.cc:1918] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:50:09.033679 14227 log.cc:1196] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/d12a0b9d82d94e4ab9c2b297a49af8bf
I20260928 23:50:09.033684 14228 ts_tablet_manager.cc:1939] T d12a0b9d82d94e4ab9c2b297a49af8bf P 90b865839cc84946afb5399f87beee52: Deleting consensus metadata
I20260928 23:50:09.033931 14227 ts_tablet_manager.cc:1939] T d12a0b9d82d94e4ab9c2b297a49af8bf P 24026d9eea29427bb69abcf470ecf1da: Deleting consensus metadata
I20260928 23:50:09.034525 13643 catalog_manager.cc:4841] TS 90b865839cc84946afb5399f87beee52 (127.6.152.2:46775): tablet d12a0b9d82d94e4ab9c2b297a49af8bf (table test-table [id=1db250a8025b4ffe976c3f681ed660f5]) successfully deleted
I20260928 23:50:09.034737 13643 catalog_manager.cc:4841] TS 24026d9eea29427bb69abcf470ecf1da (127.6.152.3:33437): tablet d12a0b9d82d94e4ab9c2b297a49af8bf (table test-table [id=1db250a8025b4ffe976c3f681ed660f5]) successfully deleted
I20260928 23:50:09.039134 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 13846
I20260928 23:50:09.039645 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:09.039298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:38171 (local address 127.0.0.1:50584)
0928 23:50:09.039493 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:09.039547 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.1:38171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260928 23:50:09.039885 13644 catalog_manager.cc:4584] TS af5a3b2b50fa4554a12490c58ccbb825 (127.6.152.1:38171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d12a0b9d82d94e4ab9c2b297a49af8bf: Network error: Client connection negotiation failed: client connection to 127.6.152.1:38171: connect: Connection refused (error 111)
I20260928 23:50:09.045552 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 13983
I20260928 23:50:09.052147 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 13330
I20260928 23:50:09.057935 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 13437
2026-09-28T23:50:09Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5595 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:50:09 dist-test-slave-p5bw krb5kdc[14232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:50:09 dist-test-slave-p5bw krb5kdc[14232](info): set up 2 sockets
Sep 28 23:50:09 dist-test-slave-p5bw krb5kdc[14232](info): commencing operation
krb5kdc: starting...
W20260928 23:50:10.098155 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:50:10 dist-test-slave-p5bw krb5kdc[14232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:50:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:50:10Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.62.keytab.
Entry for principal oryx/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.62.keytab.
I20260928 23:50:10.216398 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:41033
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.62.keytab
--principal=oryx/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:40863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:41033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:50:10.323587 14245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:10.323870 14245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:10.323943 14245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:10.327874 14245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:50:10.327960 14245 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 23:50:10.327996 14245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:10.328056 14245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:10.328111 14245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:50:10.328146 14245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:50:10.333355 14245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:40863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:41033
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.62.keytab
--principal=oryx/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:41033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:10.334354 14245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:10.335342 14245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:10.343632 14250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:10.343626 14251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:10.343631 14253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:10.343676 14245 server_base.cc:1034] running on GCE node
I20260928 23:50:10.344465 14245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:10.345606 14245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:10.346767 14245 hybrid_clock.cc:648] HybridClock initialized: now 1790639410346754 us; error 24 us; skew 500 ppm
Sep 28 23:50:10 dist-test-slave-p5bw krb5kdc[14232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639410, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:10.349440 14245 init.cc:377] Logged in from keytab as oryx/127.6.152.62@KRBTEST.COM (short username oryx)
I20260928 23:50:10.350319 14245 webserver.cc:460] Webserver started at http://127.6.152.62:43303/ using document root <none> and password file <none>
I20260928 23:50:10.350863 14245 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:10.350920 14245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:10.351142 14245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:10.353178 14245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "decdf3ad9efc41159cb3e3ef18bae1fa"
format_stamp: "Formatted at 2026-09-28 23:50:10 on dist-test-slave-p5bw"
server_key: "4eed1efd09e7b43f7efe8642532cdcd6"
server_key_iv: "aecb3aa8a071ded9f17e5242d12a5423"
server_key_version: "encryptionkey@0"
I20260928 23:50:10.353724 14245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "decdf3ad9efc41159cb3e3ef18bae1fa"
format_stamp: "Formatted at 2026-09-28 23:50:10 on dist-test-slave-p5bw"
server_key: "4eed1efd09e7b43f7efe8642532cdcd6"
server_key_iv: "aecb3aa8a071ded9f17e5242d12a5423"
server_key_version: "encryptionkey@0"
I20260928 23:50:10.357262 14245 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:50:10.359476 14260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:10.360651 14245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:50:10.360786 14245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "decdf3ad9efc41159cb3e3ef18bae1fa"
format_stamp: "Formatted at 2026-09-28 23:50:10 on dist-test-slave-p5bw"
server_key: "4eed1efd09e7b43f7efe8642532cdcd6"
server_key_iv: "aecb3aa8a071ded9f17e5242d12a5423"
server_key_version: "encryptionkey@0"
I20260928 23:50:10.360956 14245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:10.382302 14245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:10.385862 14245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:10.386052 14245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:10.393613 14245 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:41033
I20260928 23:50:10.393639 14322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:41033 every 8 connection(s)
I20260928 23:50:10.394554 14245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:50:10.397365 14323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:10.402182 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 14245
I20260928 23:50:10.402328 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:50:10.402592 6752 external_mini_cluster.cc:1428] Setting key 64c734d723cd9e1554d4ac687906f6fc
I20260928 23:50:10.402621 14323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa: Bootstrap starting.
I20260928 23:50:10.405136 14323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:10.405975 14323 log.cc:824] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa: Log is configured to *not* fsync() on all Append() calls
I20260928 23:50:10.408067 14323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa: No bootstrap required, opened a new log
Sep 28 23:50:10 dist-test-slave-p5bw krb5kdc[14232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM
I20260928 23:50:10.412392 14323 raft_consensus.cc:348] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "decdf3ad9efc41159cb3e3ef18bae1fa" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41033 } }
I20260928 23:50:10.412628 14323 raft_consensus.cc:374] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:10.412696 14323 raft_consensus.cc:729] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: decdf3ad9efc41159cb3e3ef18bae1fa, State: Initialized, Role: FOLLOWER
I20260928 23:50:10.413173 14323 consensus_queue.cc:260] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "decdf3ad9efc41159cb3e3ef18bae1fa" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41033 } }
I20260928 23:50:10.413357 14323 raft_consensus.cc:388] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:50:10.413437 14323 raft_consensus.cc:482] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:50:10.413549 14323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:10.414618 14323 raft_consensus.cc:504] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "decdf3ad9efc41159cb3e3ef18bae1fa" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41033 } }
I20260928 23:50:10.414966 14323 leader_election.cc:302] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: decdf3ad9efc41159cb3e3ef18bae1fa; no voters:
I20260928 23:50:10.415328 14323 leader_election.cc:288] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:50:10.415397 14328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:50:10.415618 14328 raft_consensus.cc:686] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa [term 1 LEADER]: Becoming Leader. State: Replica: decdf3ad9efc41159cb3e3ef18bae1fa, State: Running, Role: LEADER
I20260928 23:50:10.415910 14328 consensus_queue.cc:237] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "decdf3ad9efc41159cb3e3ef18bae1fa" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41033 } }
I20260928 23:50:10.416659 14323 sys_catalog.cc:564] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:50:10.417085 14328 sys_catalog.cc:455] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader decdf3ad9efc41159cb3e3ef18bae1fa. Latest consensus state: current_term: 1 leader_uuid: "decdf3ad9efc41159cb3e3ef18bae1fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "decdf3ad9efc41159cb3e3ef18bae1fa" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41033 } } }
I20260928 23:50:10.417197 14328 sys_catalog.cc:458] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:10.417536 14332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:50:10.418220 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:10.403910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46386 (local address 127.6.152.62:41033)
0928 23:50:10.404309 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:10.404327 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:10.404366 (+ 39us) server_negotiation.cc:407] Connection header received
0928 23:50:10.405254 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:10.405278 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:10.405700 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:10.406053 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:10.406899 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:10.408292 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:10.409149 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:10.409930 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:10.410102 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:10.412893 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:10.412898 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:10.412911 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:10.412940 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:10.414829 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:10.415299 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:10.415305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:10.415311 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:10.415507 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:10.415873 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:10.415877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:10.415878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:10.416343 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:10.416542 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:10.416869 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 23:50:10.417201 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":117,"threads_started":1}
I20260928 23:50:10.420070 14330 sys_catalog.cc:455] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "decdf3ad9efc41159cb3e3ef18bae1fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "decdf3ad9efc41159cb3e3ef18bae1fa" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41033 } } }
I20260928 23:50:10.420171 14330 sys_catalog.cc:458] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:10.420969 14332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:50:10.426507 14332 catalog_manager.cc:1347] Generated new cluster ID: 982f624063424d02b7299e8b46280fc5
I20260928 23:50:10.426584 14332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:50:10.437240 14332 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:50:10.438421 14332 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:50:10.444490 14332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa: Generated new TSK 0
I20260928 23:50:10.445189 14332 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.1.keytab.
Entry for principal oryx/127.6.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.1.keytab.
I20260928 23:50:10.473609 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:41033
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.1.keytab
--principal=oryx/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:40863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:50:10.581414 14349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:10.581714 14349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:10.581790 14349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:10.585804 14349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:10.585891 14349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:10.585959 14349 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:50:10.586045 14349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:50:10.591305 14349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:40863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.1.keytab
--principal=oryx/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.152.62:41033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:10.592392 14349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:10.593425 14349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:10.603068 14354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:10.603086 14357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:10.603101 14355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:10.603408 14349 server_base.cc:1034] running on GCE node
I20260928 23:50:10.603832 14349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:10.604539 14349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:10.605729 14349 hybrid_clock.cc:648] HybridClock initialized: now 1790639410605678 us; error 59 us; skew 500 ppm
Sep 28 23:50:10 dist-test-slave-p5bw krb5kdc[14232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639410, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:10.608529 14349 init.cc:377] Logged in from keytab as oryx/127.6.152.1@KRBTEST.COM (short username oryx)
I20260928 23:50:10.609426 14349 webserver.cc:460] Webserver started at http://127.6.152.1:34325/ using document root <none> and password file <none>
I20260928 23:50:10.609982 14349 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:10.610044 14349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:10.610251 14349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:10.612365 14349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "38f20112572b4d3591be33a57fefca51"
format_stamp: "Formatted at 2026-09-28 23:50:10 on dist-test-slave-p5bw"
server_key: "a47e830010472d09f3cdb884d1bd955f"
server_key_iv: "852a9683656dbbf57492e4298ca99db8"
server_key_version: "encryptionkey@0"
I20260928 23:50:10.612991 14349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "38f20112572b4d3591be33a57fefca51"
format_stamp: "Formatted at 2026-09-28 23:50:10 on dist-test-slave-p5bw"
server_key: "a47e830010472d09f3cdb884d1bd955f"
server_key_iv: "852a9683656dbbf57492e4298ca99db8"
server_key_version: "encryptionkey@0"
I20260928 23:50:10.616660 14349 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:50:10.619318 14364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:10.620508 14349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:50:10.620628 14349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "38f20112572b4d3591be33a57fefca51"
format_stamp: "Formatted at 2026-09-28 23:50:10 on dist-test-slave-p5bw"
server_key: "a47e830010472d09f3cdb884d1bd955f"
server_key_iv: "852a9683656dbbf57492e4298ca99db8"
server_key_version: "encryptionkey@0"
I20260928 23:50:10.620734 14349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:10.635589 14349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:10.639060 14349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:10.639271 14349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:10.640705 14349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:10.640753 14349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:10.640793 14349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:10.640875 14349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:50:10 dist-test-slave-p5bw krb5kdc[14232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639410, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM
I20260928 23:50:10.651937 14349 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:34437
I20260928 23:50:10.651999 14479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:34437 every 8 connection(s)
I20260928 23:50:10.652978 14349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:50:10.654875 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:10.641686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:51551 (local address 127.6.152.62:41033)
0928 23:50:10.641867 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:10.641872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:10.642658 (+ 786us) server_negotiation.cc:407] Connection header received
0928 23:50:10.643698 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:10.643702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:10.643768 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:10.643814 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:10.645643 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:10.646323 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:10.647698 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:10.647993 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:10.648108 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:10.651179 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:10.651181 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:10.651184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:10.651210 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:10.652784 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:10.653400 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:10.653404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:10.653405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:10.653504 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:10.653880 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:10.653884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:10.653885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:10.654140 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:10.654235 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:10.654660 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 23:50:10.654761 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 23:50:10.655627 14373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:10.642046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:41033 (local address 127.6.152.1:51551)
0928 23:50:10.642488 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:10.642536 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:10.643420 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:10.644295 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:10.644305 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:10.644859 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:10.645500 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:10.645516 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:10.646442 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:10.646447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:10.647576 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:10.647586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:10.648172 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:10.648177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:10.648399 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:10.649086 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:10.649102 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:10.651014 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:10.652941 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:10.652950 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:10.652973 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:10.653297 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:10.653615 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:10.653618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:10.653620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:10.653757 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:10.654230 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:10.654235 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:10.654447 (+ 212us) client_negotiation.cc:769] Sending connection context
0928 23:50:10.654607 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 23:50:10.654841 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":128,"threads_started":1}
I20260928 23:50:10.656967 14480 heartbeater.cc:347] Connected to a master server at 127.6.152.62:41033
I20260928 23:50:10.657123 14480 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:10.657614 14480 heartbeater.cc:511] Master 127.6.152.62:41033 requested a full tablet report, sending...
I20260928 23:50:10.659198 14277 ts_manager.cc:194] Registered new tserver with Master: 38f20112572b4d3591be33a57fefca51 (127.6.152.1:34437)
I20260928 23:50:10.659556 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 14349
I20260928 23:50:10.659687 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
I20260928 23:50:10.660005 6752 external_mini_cluster.cc:1428] Setting key 8e54a92a3a6d0723d9e792aefb97bf75
I20260928 23:50:10.660723 14277 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.152.1@KRBTEST.COM'} at 127.6.152.1:51551
I20260928 23:50:10.668291 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:10.659107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:58321 (local address 127.6.152.62:41033)
0928 23:50:10.659210 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:10.659216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:10.660525 (+ 1309us) server_negotiation.cc:407] Connection header received
0928 23:50:10.660622 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:10.660625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:10.660674 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:10.660784 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:10.661315 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:10.662115 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:10.663450 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:10.663822 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:10.663940 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:10.665177 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:10.665183 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:10.665185 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:10.665206 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:10.666686 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:10.667171 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:10.667173 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:10.667174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:10.667244 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:10.667559 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:10.667561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:10.667562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:10.667706 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:10.667777 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:10.668096 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 23:50:10.668182 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 23:50:10.668452 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:10.659039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:41033 (local address 127.6.152.1:58321)
0928 23:50:10.660426 (+ 1387us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:10.660441 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:10.660535 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:10.660802 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:10.660805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:10.661097 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:10.661196 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:10.661202 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:10.662261 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:10.662268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:10.663299 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:10.663313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:10.663930 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:10.663932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:10.664031 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:10.664599 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:10.664619 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:10.665020 (+ 401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:10.666796 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:10.666803 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:10.666808 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:10.667071 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:10.667336 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:10.667340 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:10.667345 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:10.667463 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:10.667801 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:10.667806 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:10.667922 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 23:50:10.668050 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 23:50:10.668218 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for oryx/127.6.152.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.152.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.152.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.2.keytab.
Entry for principal oryx/127.6.152.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.2.keytab.
I20260928 23:50:10.688405 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:41033
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.2.keytab
--principal=oryx/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:40863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:50:10.799531 14489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:10.799783 14489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:10.799829 14489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:10.803728 14489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:10.803776 14489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:10.803814 14489 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:50:10.803844 14489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:50:10.808799 14489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:40863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.2.keytab
--principal=oryx/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.152.62:41033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:10.809739 14489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:10.810659 14489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:10.820338 14494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:10.820351 14495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:10.820371 14489 server_base.cc:1034] running on GCE node
W20260928 23:50:10.820370 14497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:10.821059 14489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:10.821660 14489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:10.822878 14489 hybrid_clock.cc:648] HybridClock initialized: now 1790639410822862 us; error 32 us; skew 500 ppm
Sep 28 23:50:10 dist-test-slave-p5bw krb5kdc[14232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639410, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:10.825850 14489 init.cc:377] Logged in from keytab as oryx/127.6.152.2@KRBTEST.COM (short username oryx)
I20260928 23:50:10.826623 14489 webserver.cc:460] Webserver started at http://127.6.152.2:34151/ using document root <none> and password file <none>
I20260928 23:50:10.827111 14489 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:10.827169 14489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:10.827325 14489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:10.829288 14489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "a29624e230b7459a9ca2012108f5a54a"
format_stamp: "Formatted at 2026-09-28 23:50:10 on dist-test-slave-p5bw"
server_key: "6e22b7acd013b247c6c12792c046c726"
server_key_iv: "118337948fccb7eb37645bcca5df9c60"
server_key_version: "encryptionkey@0"
I20260928 23:50:10.829779 14489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "a29624e230b7459a9ca2012108f5a54a"
format_stamp: "Formatted at 2026-09-28 23:50:10 on dist-test-slave-p5bw"
server_key: "6e22b7acd013b247c6c12792c046c726"
server_key_iv: "118337948fccb7eb37645bcca5df9c60"
server_key_version: "encryptionkey@0"
I20260928 23:50:10.833287 14489 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:50:10.835601 14504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:10.836800 14489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:50:10.837008 14489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "a29624e230b7459a9ca2012108f5a54a"
format_stamp: "Formatted at 2026-09-28 23:50:10 on dist-test-slave-p5bw"
server_key: "6e22b7acd013b247c6c12792c046c726"
server_key_iv: "118337948fccb7eb37645bcca5df9c60"
server_key_version: "encryptionkey@0"
I20260928 23:50:10.837107 14489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:10.862329 14489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:10.865796 14489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:10.865996 14489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:10.867403 14489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:10.867448 14489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:10.867514 14489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:10.867545 14489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:50:10 dist-test-slave-p5bw krb5kdc[14232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639410, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM
I20260928 23:50:10.878371 14489 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:38143
I20260928 23:50:10.878494 14619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:38143 every 8 connection(s)
I20260928 23:50:10.879510 14489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:50:10.881482 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:10.868242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:38201 (local address 127.6.152.62:41033)
0928 23:50:10.868368 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:10.868372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:10.869454 (+ 1082us) server_negotiation.cc:407] Connection header received
0928 23:50:10.870547 (+ 1093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:10.870550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:10.870631 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:10.870697 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:10.872471 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:10.873407 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:10.874863 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:10.875097 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:10.875211 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:10.877703 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:10.877709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:10.877715 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:10.877751 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:10.879439 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:10.880002 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:10.880005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:10.880007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:10.880093 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:10.880438 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:10.880440 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:10.880441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:10.880621 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:10.880695 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:10.881235 (+ 540us) server_negotiation.cc:299] Negotiation successful
0928 23:50:10.881367 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:50:10.882222 14513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:10.868839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:41033 (local address 127.6.152.2:38201)
0928 23:50:10.869318 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:10.869359 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:10.870288 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:10.871154 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:10.871166 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:10.871578 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:10.872283 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:10.872301 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:10.873550 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:10.873554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:10.874738 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:10.874747 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:10.875249 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:10.875253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:10.875427 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:10.875988 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:10.876003 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:10.877522 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:10.879548 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:10.879577 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:10.879592 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:10.879884 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:10.880173 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:10.880176 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:10.880179 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:10.880342 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:10.880713 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:10.880718 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:10.881033 (+ 315us) client_negotiation.cc:769] Sending connection context
0928 23:50:10.881188 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 23:50:10.881446 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":135,"threads_started":1}
I20260928 23:50:10.883548 14620 heartbeater.cc:347] Connected to a master server at 127.6.152.62:41033
I20260928 23:50:10.883701 14620 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:10.884182 14620 heartbeater.cc:511] Master 127.6.152.62:41033 requested a full tablet report, sending...
I20260928 23:50:10.884706 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 14489
I20260928 23:50:10.884869 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:50:10.885185 6752 external_mini_cluster.cc:1428] Setting key 44089d86fa39986deceb0db8ea6ced0c
I20260928 23:50:10.885264 14272 ts_manager.cc:194] Registered new tserver with Master: a29624e230b7459a9ca2012108f5a54a (127.6.152.2:38143)
I20260928 23:50:10.886001 14272 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.152.2@KRBTEST.COM'} at 127.6.152.2:38201
I20260928 23:50:10.893980 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:10.885918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:41033 (local address 127.6.152.2:38903)
0928 23:50:10.886256 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:10.886271 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:10.886359 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:10.886612 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:10.886615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:10.886822 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:10.886943 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:10.886950 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:10.887983 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:10.887986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:10.888945 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:10.888957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:10.889476 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:10.889483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:10.889617 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:10.890071 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:10.890090 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:10.890524 (+ 434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:10.892450 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:10.892453 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:10.892454 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:10.892634 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:10.892952 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:10.892954 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:10.892955 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:10.893033 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:10.893533 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:10.893536 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:10.893612 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 23:50:10.893715 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 23:50:10.893819 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":98,"threads_started":1}
I20260928 23:50:10.894097 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:10.885840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:38903 (local address 127.6.152.62:41033)
0928 23:50:10.886339 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:10.886343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:10.886367 (+ 24us) server_negotiation.cc:407] Connection header received
0928 23:50:10.886449 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:10.886454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:10.886503 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:10.886580 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:10.887083 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:10.887807 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:10.889152 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:10.889359 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:10.889445 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:10.890654 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:10.890658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:10.890660 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:10.890680 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:10.892333 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:10.892732 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:10.892737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:10.892741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:10.892861 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:10.893124 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:10.893128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:10.893133 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:10.893415 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:10.893527 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:10.893795 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 23:50:10.893916 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427}
WARNING: no policy specified for oryx/127.6.152.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.152.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.152.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.3.keytab.
Entry for principal oryx/127.6.152.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.3.keytab.
I20260928 23:50:10.911502 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:41033
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.3.keytab
--principal=oryx/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:40863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:50:11.020215 14629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:11.020445 14629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:11.020512 14629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:11.024492 14629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:11.024542 14629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:11.024608 14629 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:50:11.024698 14629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:50:11.029868 14629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:40863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.3.keytab
--principal=oryx/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.152.62:41033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:11.030872 14629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:11.031867 14629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:11.041213 14635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:11.041270 14629 server_base.cc:1034] running on GCE node
W20260928 23:50:11.041224 14634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:11.041224 14637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:11.042035 14629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:11.042629 14629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:11.043803 14629 hybrid_clock.cc:648] HybridClock initialized: now 1790639411043769 us; error 45 us; skew 500 ppm
Sep 28 23:50:11 dist-test-slave-p5bw krb5kdc[14232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639411, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:11.046412 14629 init.cc:377] Logged in from keytab as oryx/127.6.152.3@KRBTEST.COM (short username oryx)
I20260928 23:50:11.047259 14629 webserver.cc:460] Webserver started at http://127.6.152.3:40993/ using document root <none> and password file <none>
I20260928 23:50:11.047798 14629 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:11.047885 14629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:11.048102 14629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:11.050055 14629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "e875b6bbdcf849538e3d71b22af61384"
format_stamp: "Formatted at 2026-09-28 23:50:11 on dist-test-slave-p5bw"
server_key: "f9e13d74d55bd64f483bdf2a940d7545"
server_key_iv: "b8cda1cefc9fbd9e5404ebd882a3e9e8"
server_key_version: "encryptionkey@0"
I20260928 23:50:11.050603 14629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "e875b6bbdcf849538e3d71b22af61384"
format_stamp: "Formatted at 2026-09-28 23:50:11 on dist-test-slave-p5bw"
server_key: "f9e13d74d55bd64f483bdf2a940d7545"
server_key_iv: "b8cda1cefc9fbd9e5404ebd882a3e9e8"
server_key_version: "encryptionkey@0"
I20260928 23:50:11.054138 14629 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:50:11.056496 14644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:11.057909 14629 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:50:11.058027 14629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "e875b6bbdcf849538e3d71b22af61384"
format_stamp: "Formatted at 2026-09-28 23:50:11 on dist-test-slave-p5bw"
server_key: "f9e13d74d55bd64f483bdf2a940d7545"
server_key_iv: "b8cda1cefc9fbd9e5404ebd882a3e9e8"
server_key_version: "encryptionkey@0"
I20260928 23:50:11.058125 14629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:11.082870 14629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:11.085999 14629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:11.086198 14629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:11.087693 14629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:11.087740 14629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:11.087805 14629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:11.087839 14629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:50:11 dist-test-slave-p5bw krb5kdc[14232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639411, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM
I20260928 23:50:11.098660 14629 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:46097
I20260928 23:50:11.098771 14759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:46097 every 8 connection(s)
I20260928 23:50:11.099807 14629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:50:11.102010 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.088502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:56645 (local address 127.6.152.62:41033)
0928 23:50:11.088679 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:11.088686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:11.089587 (+ 901us) server_negotiation.cc:407] Connection header received
0928 23:50:11.090912 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:11.090915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:11.090992 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:11.091104 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:11.092614 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.093532 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.094860 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.095082 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.095195 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.098194 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:11.098203 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:11.098205 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:11.098228 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:11.100014 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:11.100591 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:11.100600 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:11.100601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:11.100705 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:11.101101 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:11.101103 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:11.101105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:11.101282 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:11.101362 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:11.101785 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 23:50:11.101886 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 23:50:11.102810 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.088915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:41033 (local address 127.6.152.3:56645)
0928 23:50:11.089408 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:11.089469 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:11.090615 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:11.091301 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:11.091311 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:11.091822 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:11.092449 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.092470 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.093669 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.093673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.094734 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.094742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.095239 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.095243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.095407 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.096442 (+ 1035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:11.096457 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:11.098011 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:11.100124 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:11.100147 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:11.100163 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:11.100482 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:11.100791 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:11.100798 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:11.100800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:11.100985 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:11.101368 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:11.101372 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:11.101583 (+ 211us) client_negotiation.cc:769] Sending connection context
0928 23:50:11.101721 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 23:50:11.101946 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":107,"threads_started":1}
I20260928 23:50:11.103971 14760 heartbeater.cc:347] Connected to a master server at 127.6.152.62:41033
I20260928 23:50:11.104121 14760 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:11.104573 14760 heartbeater.cc:511] Master 127.6.152.62:41033 requested a full tablet report, sending...
I20260928 23:50:11.105576 14272 ts_manager.cc:194] Registered new tserver with Master: e875b6bbdcf849538e3d71b22af61384 (127.6.152.3:46097)
I20260928 23:50:11.106138 14272 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.152.3@KRBTEST.COM'} at 127.6.152.3:56645
I20260928 23:50:11.107617 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 14629
I20260928 23:50:11.107726 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:50:11.108104 6752 external_mini_cluster.cc:1428] Setting key d3cb175eff71fc656211f500be275f6f
I20260928 23:50:11.110054 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:50:11.113627 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.106232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:55749 (local address 127.6.152.62:41033)
0928 23:50:11.106361 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:11.106366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:11.106547 (+ 181us) server_negotiation.cc:407] Connection header received
0928 23:50:11.106658 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:11.106660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:11.106694 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:11.106753 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:11.107220 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.107941 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.109022 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.109283 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.109415 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.110528 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:11.110530 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:11.110533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:11.110552 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:11.111944 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:11.112483 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:11.112486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:11.112488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:11.112580 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:11.112897 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:11.112901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:11.112907 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:11.113059 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:11.113138 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:11.113427 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 23:50:11.113511 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 23:50:11.113842 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.106172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:41033 (local address 127.6.152.3:55749)
0928 23:50:11.106463 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:11.106477 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:11.106578 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:11.106791 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:11.106794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:11.106990 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:11.107092 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.107099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.108090 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.108093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.108922 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.108932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.109401 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.109404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.109499 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.109984 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:11.110005 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:11.110397 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:11.112045 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:11.112050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:11.112054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:11.112316 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:11.112677 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:11.112681 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:11.112684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:11.112790 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:11.113139 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:11.113145 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:11.113261 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 23:50:11.113377 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 23:50:11.113543 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":117,"threads_started":1}
Sep 28 23:50:11 dist-test-slave-p5bw krb5kdc[14232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639411, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 23:50:11 dist-test-slave-p5bw krb5kdc[14232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:11 dist-test-slave-p5bw krb5kdc[14232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 23:50:11.133795 14326 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:11.127712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46394 (local address 127.6.152.62:41033)
0928 23:50:11.127832 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:11.127836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:11.127848 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:50:11.127883 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:11.127885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:11.127921 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:11.127992 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:11.128649 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.129226 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.130125 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.130345 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.130435 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.133633 (+ 3198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46394: BlockingRecv error: recv got EOF from 127.0.0.1:46394 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 23:50:11 dist-test-slave-p5bw krb5kdc[14232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639411, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM
I20260928 23:50:11.143291 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.135122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46404 (local address 127.6.152.62:41033)
0928 23:50:11.135231 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:11.135234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:11.135360 (+ 126us) server_negotiation.cc:407] Connection header received
0928 23:50:11.135458 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:11.135460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:11.135494 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:11.135557 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:11.136142 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.136609 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.137427 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.137630 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.137702 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.139973 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:11.139976 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:11.139978 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:11.140001 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:11.141763 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:11.142158 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:11.142160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:11.142162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:11.142237 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:11.142535 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:11.142538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:11.142539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:11.142720 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:11.142830 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:11.143089 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 23:50:11.143179 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 23:50:11.146072 14272 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:50:11.149072 14272 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:50:11.162729 14784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.157208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:34437 (local address 127.0.0.1:60558)
0928 23:50:11.157918 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:11.157968 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:11.158100 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:11.158863 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:11.158869 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:11.158891 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:11.159014 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.159020 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.160111 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.160116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.161387 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.161399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.162206 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.162210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.162317 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.162362 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 23:50:11.162466 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 23:50:11.162588 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":72,"threads_started":1}
I20260928 23:50:11.162781 14783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.157370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60558 (local address 127.6.152.1:34437)
0928 23:50:11.158374 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:11.158383 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:11.158410 (+ 27us) server_negotiation.cc:407] Connection header received
0928 23:50:11.158489 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:11.158494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:11.158718 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:11.158875 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:11.159138 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.160000 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.161511 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.162073 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.162208 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.162491 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:11.162547 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 23:50:11.162637 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":899,"thread_start_us":97,"threads_started":1}
I20260928 23:50:11.163780 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.157879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:38143 (local address 127.0.0.1:46704)
0928 23:50:11.158541 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:11.158556 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:11.158650 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:11.159172 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:11.159175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:11.159190 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:11.159264 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.159270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.160652 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.160658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.162099 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.162110 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.163348 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.163351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.163455 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.163493 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 23:50:11.163585 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 23:50:11.163655 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":232,"thread_start_us":94,"threads_started":1}
I20260928 23:50:11.163836 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.158288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46704 (local address 127.6.152.2:38143)
0928 23:50:11.158723 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:11.158735 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:11.158758 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:50:11.158852 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:11.158857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:11.159055 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:11.159200 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:11.159349 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.160520 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.162238 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.163057 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.163151 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.163287 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:11.163618 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 23:50:11.163706 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":107,"threads_started":1}
I20260928 23:50:11.163961 14786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.157935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42096 (local address 127.6.152.3:46097)
0928 23:50:11.158382 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:11.158394 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:11.158579 (+ 185us) server_negotiation.cc:407] Connection header received
0928 23:50:11.158775 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:11.158783 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:11.158947 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:11.159059 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:11.159485 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.160592 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.162289 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.163052 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.163137 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.163287 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:11.163667 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 23:50:11.163775 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":100,"threads_started":1}
I20260928 23:50:11.164294 14782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.157401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:46097 (local address 127.0.0.1:42096)
0928 23:50:11.158501 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:11.158513 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:11.158638 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:11.159250 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:11.159252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:11.159270 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:11.159362 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.159368 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.160700 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.160703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.162171 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.162181 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.163331 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.163334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.163452 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.163496 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 23:50:11.164100 (+ 604us) client_negotiation.cc:240] Negotiation successful
0928 23:50:11.164178 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1043,"mutex_wait_us":50,"thread_start_us":86,"threads_started":1}
I20260928 23:50:11.166038 14414 tablet_service.cc:1467] Processing CreateTablet for tablet 4de82d2f6bd44c24b874e54c10b86865 (DEFAULT_TABLE table=test-table [id=f889432c093742878079c6d6d8a89d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:50:11.166152 14554 tablet_service.cc:1467] Processing CreateTablet for tablet 4de82d2f6bd44c24b874e54c10b86865 (DEFAULT_TABLE table=test-table [id=f889432c093742878079c6d6d8a89d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:50:11.167008 14554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4de82d2f6bd44c24b874e54c10b86865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:11.167048 14414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4de82d2f6bd44c24b874e54c10b86865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:11.168926 14694 tablet_service.cc:1467] Processing CreateTablet for tablet 4de82d2f6bd44c24b874e54c10b86865 (DEFAULT_TABLE table=test-table [id=f889432c093742878079c6d6d8a89d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:50:11.169863 14694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4de82d2f6bd44c24b874e54c10b86865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:11.172922 14788 tablet_bootstrap.cc:492] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51: Bootstrap starting.
I20260928 23:50:11.175002 14789 tablet_bootstrap.cc:492] T 4de82d2f6bd44c24b874e54c10b86865 P a29624e230b7459a9ca2012108f5a54a: Bootstrap starting.
I20260928 23:50:11.175896 14788 tablet_bootstrap.cc:654] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:11.176745 14788 log.cc:824] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51: Log is configured to *not* fsync() on all Append() calls
I20260928 23:50:11.176913 14790 tablet_bootstrap.cc:492] T 4de82d2f6bd44c24b874e54c10b86865 P e875b6bbdcf849538e3d71b22af61384: Bootstrap starting.
I20260928 23:50:11.177155 14789 tablet_bootstrap.cc:654] T 4de82d2f6bd44c24b874e54c10b86865 P a29624e230b7459a9ca2012108f5a54a: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:11.177979 14789 log.cc:824] T 4de82d2f6bd44c24b874e54c10b86865 P a29624e230b7459a9ca2012108f5a54a: Log is configured to *not* fsync() on all Append() calls
I20260928 23:50:11.178948 14790 tablet_bootstrap.cc:654] T 4de82d2f6bd44c24b874e54c10b86865 P e875b6bbdcf849538e3d71b22af61384: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:11.179072 14788 tablet_bootstrap.cc:492] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51: No bootstrap required, opened a new log
I20260928 23:50:11.179262 14788 ts_tablet_manager.cc:1397] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 23:50:11.179745 14790 log.cc:824] T 4de82d2f6bd44c24b874e54c10b86865 P e875b6bbdcf849538e3d71b22af61384: Log is configured to *not* fsync() on all Append() calls
I20260928 23:50:11.179970 14789 tablet_bootstrap.cc:492] T 4de82d2f6bd44c24b874e54c10b86865 P a29624e230b7459a9ca2012108f5a54a: No bootstrap required, opened a new log
I20260928 23:50:11.180153 14789 ts_tablet_manager.cc:1397] T 4de82d2f6bd44c24b874e54c10b86865 P a29624e230b7459a9ca2012108f5a54a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 23:50:11.181600 14790 tablet_bootstrap.cc:492] T 4de82d2f6bd44c24b874e54c10b86865 P e875b6bbdcf849538e3d71b22af61384: No bootstrap required, opened a new log
I20260928 23:50:11.181773 14790 ts_tablet_manager.cc:1397] T 4de82d2f6bd44c24b874e54c10b86865 P e875b6bbdcf849538e3d71b22af61384: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 23:50:11.182920 14788 raft_consensus.cc:348] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } }
I20260928 23:50:11.183240 14788 raft_consensus.cc:374] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:11.183293 14788 raft_consensus.cc:729] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f20112572b4d3591be33a57fefca51, State: Initialized, Role: FOLLOWER
I20260928 23:50:11.183727 14788 consensus_queue.cc:260] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } }
I20260928 23:50:11.184093 14789 raft_consensus.cc:348] T 4de82d2f6bd44c24b874e54c10b86865 P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } }
I20260928 23:50:11.184343 14789 raft_consensus.cc:374] T 4de82d2f6bd44c24b874e54c10b86865 P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:11.184372 14480 heartbeater.cc:503] Master 127.6.152.62:41033 was elected leader, sending a full tablet report...
I20260928 23:50:11.184393 14789 raft_consensus.cc:729] T 4de82d2f6bd44c24b874e54c10b86865 P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29624e230b7459a9ca2012108f5a54a, State: Initialized, Role: FOLLOWER
I20260928 23:50:11.184718 14788 ts_tablet_manager.cc:1428] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 23:50:11.184880 14789 consensus_queue.cc:260] T 4de82d2f6bd44c24b874e54c10b86865 P a29624e230b7459a9ca2012108f5a54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } }
I20260928 23:50:11.185523 14620 heartbeater.cc:503] Master 127.6.152.62:41033 was elected leader, sending a full tablet report...
I20260928 23:50:11.185595 14790 raft_consensus.cc:348] T 4de82d2f6bd44c24b874e54c10b86865 P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } }
I20260928 23:50:11.185828 14790 raft_consensus.cc:374] T 4de82d2f6bd44c24b874e54c10b86865 P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:11.185892 14790 raft_consensus.cc:729] T 4de82d2f6bd44c24b874e54c10b86865 P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e875b6bbdcf849538e3d71b22af61384, State: Initialized, Role: FOLLOWER
I20260928 23:50:11.186285 14790 consensus_queue.cc:260] T 4de82d2f6bd44c24b874e54c10b86865 P e875b6bbdcf849538e3d71b22af61384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } }
I20260928 23:50:11.187209 14789 ts_tablet_manager.cc:1428] T 4de82d2f6bd44c24b874e54c10b86865 P a29624e230b7459a9ca2012108f5a54a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 23:50:11.187255 14790 ts_tablet_manager.cc:1428] T 4de82d2f6bd44c24b874e54c10b86865 P e875b6bbdcf849538e3d71b22af61384: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 23:50:11.188616 14760 heartbeater.cc:503] Master 127.6.152.62:41033 was elected leader, sending a full tablet report...
I20260928 23:50:11.304858 14794 raft_consensus.cc:482] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:50:11.305114 14794 raft_consensus.cc:504] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } }
I20260928 23:50:11.306366 14794 leader_election.cc:288] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e875b6bbdcf849538e3d71b22af61384 (127.6.152.3:46097), a29624e230b7459a9ca2012108f5a54a (127.6.152.2:38143)
I20260928 23:50:11.310621 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.306775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:37557 (local address 127.6.152.2:38143)
0928 23:50:11.306957 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:11.306961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:11.307182 (+ 221us) server_negotiation.cc:407] Connection header received
0928 23:50:11.307287 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:11.307290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:11.307330 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:11.307417 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:11.307706 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.308526 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.309650 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.310058 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.310138 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.310231 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:11.310411 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 23:50:11.310490 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 23:50:11.310649 14798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.306712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:38143 (local address 127.6.152.1:37557)
0928 23:50:11.307048 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:11.307059 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:11.307187 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:11.307429 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:11.307432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:11.307457 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:11.307585 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.307591 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.308625 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.308628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.309520 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.309529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.310176 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.310181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.310272 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.310302 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 23:50:11.310375 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 23:50:11.310450 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":36,"spinlock_wait_cycles":13312,"thread_start_us":109,"threads_started":1}
I20260928 23:50:11.310680 14786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.306703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:44907 (local address 127.6.152.3:46097)
0928 23:50:11.306832 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:11.306837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:11.307112 (+ 275us) server_negotiation.cc:407] Connection header received
0928 23:50:11.307268 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:11.307274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:11.307331 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:11.307421 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:11.307761 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.308542 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.309682 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.310085 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.310209 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.310272 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:11.310487 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 23:50:11.310566 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:50:11.310861 14797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.306591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:46097 (local address 127.6.152.1:44907)
0928 23:50:11.306991 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:11.307013 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:11.307128 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:11.307432 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:11.307435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:11.307464 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:11.307607 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.307612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.308668 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.308671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.309531 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.309540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.310231 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.310235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.310336 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.310379 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 23:50:11.310456 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 23:50:11.310514 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":23,"spinlock_wait_cycles":5376,"thread_start_us":157,"threads_started":1}
I20260928 23:50:11.311420 14574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de82d2f6bd44c24b874e54c10b86865" candidate_uuid: "38f20112572b4d3591be33a57fefca51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29624e230b7459a9ca2012108f5a54a" is_pre_election: true
I20260928 23:50:11.311429 14714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de82d2f6bd44c24b874e54c10b86865" candidate_uuid: "38f20112572b4d3591be33a57fefca51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e875b6bbdcf849538e3d71b22af61384" is_pre_election: true
I20260928 23:50:11.311738 14574 raft_consensus.cc:2454] T 4de82d2f6bd44c24b874e54c10b86865 P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38f20112572b4d3591be33a57fefca51 in term 0.
I20260928 23:50:11.311738 14714 raft_consensus.cc:2454] T 4de82d2f6bd44c24b874e54c10b86865 P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38f20112572b4d3591be33a57fefca51 in term 0.
I20260928 23:50:11.312269 14365 leader_election.cc:302] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 38f20112572b4d3591be33a57fefca51, a29624e230b7459a9ca2012108f5a54a, e875b6bbdcf849538e3d71b22af61384; no voters:
I20260928 23:50:11.312546 14794 raft_consensus.cc:2781] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:50:11.312671 14794 raft_consensus.cc:482] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:50:11.312733 14794 raft_consensus.cc:3037] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:11.313778 14794 raft_consensus.cc:504] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } }
I20260928 23:50:11.314081 14794 leader_election.cc:288] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51 [CANDIDATE]: Term 1 election: Requested vote from peers e875b6bbdcf849538e3d71b22af61384 (127.6.152.3:46097), a29624e230b7459a9ca2012108f5a54a (127.6.152.2:38143)
I20260928 23:50:11.314426 14574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de82d2f6bd44c24b874e54c10b86865" candidate_uuid: "38f20112572b4d3591be33a57fefca51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29624e230b7459a9ca2012108f5a54a"
I20260928 23:50:11.314426 14714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de82d2f6bd44c24b874e54c10b86865" candidate_uuid: "38f20112572b4d3591be33a57fefca51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e875b6bbdcf849538e3d71b22af61384"
I20260928 23:50:11.314550 14574 raft_consensus.cc:3037] T 4de82d2f6bd44c24b874e54c10b86865 P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:11.314550 14714 raft_consensus.cc:3037] T 4de82d2f6bd44c24b874e54c10b86865 P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:11.315824 14574 raft_consensus.cc:2454] T 4de82d2f6bd44c24b874e54c10b86865 P a29624e230b7459a9ca2012108f5a54a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38f20112572b4d3591be33a57fefca51 in term 1.
I20260928 23:50:11.315824 14714 raft_consensus.cc:2454] T 4de82d2f6bd44c24b874e54c10b86865 P e875b6bbdcf849538e3d71b22af61384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38f20112572b4d3591be33a57fefca51 in term 1.
I20260928 23:50:11.316136 14365 leader_election.cc:302] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 38f20112572b4d3591be33a57fefca51, a29624e230b7459a9ca2012108f5a54a, e875b6bbdcf849538e3d71b22af61384; no voters:
I20260928 23:50:11.316335 14794 raft_consensus.cc:2781] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:50:11.316552 14794 raft_consensus.cc:686] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51 [term 1 LEADER]: Becoming Leader. State: Replica: 38f20112572b4d3591be33a57fefca51, State: Running, Role: LEADER
I20260928 23:50:11.316996 14794 consensus_queue.cc:237] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } }
I20260928 23:50:11.320277 14276 catalog_manager.cc:5495] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38f20112572b4d3591be33a57fefca51 (127.6.152.1). New cstate: current_term: 1 leader_uuid: "38f20112572b4d3591be33a57fefca51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } health_report { overall_health: UNKNOWN } } }
W20260928 23:50:11.351066 14761 tablet.cc:2369] T 4de82d2f6bd44c24b874e54c10b86865 P e875b6bbdcf849538e3d71b22af61384: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:50:11.353966 14783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.349854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57988 (local address 127.6.152.1:34437)
0928 23:50:11.350017 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:11.350022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:11.350039 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:50:11.350239 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:11.350242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:11.350305 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:11.350440 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:50:11.350742 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.351298 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.352324 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.352547 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.352627 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.352985 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:50:11.353407 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:50:11.353492 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:11.353716 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 23:50:11.353797 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 23:50:11.360214 14714 raft_consensus.cc:1261] T 4de82d2f6bd44c24b874e54c10b86865 P e875b6bbdcf849538e3d71b22af61384 [term 1 FOLLOWER]: Refusing update from remote peer 38f20112572b4d3591be33a57fefca51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:50:11.360215 14574 raft_consensus.cc:1261] T 4de82d2f6bd44c24b874e54c10b86865 P a29624e230b7459a9ca2012108f5a54a [term 1 FOLLOWER]: Refusing update from remote peer 38f20112572b4d3591be33a57fefca51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:50:11.360988 14799 consensus_queue.cc:1035] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:50:11.361158 14794 consensus_queue.cc:1035] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:50:11.370019 14804 mvcc.cc:205] Tried to move back new op lower bound from 7334459028924039168 to 7334459028757168128. Current Snapshot: MvccSnapshot[applied={T|T < 7334459028924039168}]
I20260928 23:50:11.371970 14802 mvcc.cc:205] Tried to move back new op lower bound from 7334459028924039168 to 7334459028757168128. Current Snapshot: MvccSnapshot[applied={T|T < 7334459028924039168 or (T in {7334459028924039168})}]
I20260928 23:50:11.378824 14272 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:50:11.379061 14272 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:50:11.381492 14272 catalog_manager.cc:5782] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa: Sending DeleteTablet for 3 replicas of tablet 4de82d2f6bd44c24b874e54c10b86865
I20260928 23:50:11.382332 14694 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de82d2f6bd44c24b874e54c10b86865 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:50:11 UTC) from {username='oryx', principal='oryx/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:42096
I20260928 23:50:11.382401 14414 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de82d2f6bd44c24b874e54c10b86865 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:50:11 UTC) from {username='oryx', principal='oryx/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:60558
I20260928 23:50:11.382542 14554 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de82d2f6bd44c24b874e54c10b86865 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:50:11 UTC) from {username='oryx', principal='oryx/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:46704
I20260928 23:50:11.382977 14814 tablet_replica.cc:330] T 4de82d2f6bd44c24b874e54c10b86865 P a29624e230b7459a9ca2012108f5a54a: stopping tablet replica
I20260928 23:50:11.383011 14813 tablet_replica.cc:330] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51: stopping tablet replica
I20260928 23:50:11.383308 14814 raft_consensus.cc:2229] T 4de82d2f6bd44c24b874e54c10b86865 P a29624e230b7459a9ca2012108f5a54a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:50:11.383311 14813 raft_consensus.cc:2229] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:50:11.383656 14813 raft_consensus.cc:2258] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:50:11.383687 14814 raft_consensus.cc:2258] T 4de82d2f6bd44c24b874e54c10b86865 P a29624e230b7459a9ca2012108f5a54a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:50:11.384272 14812 tablet_replica.cc:330] T 4de82d2f6bd44c24b874e54c10b86865 P e875b6bbdcf849538e3d71b22af61384: stopping tablet replica
I20260928 23:50:11.384323 14272 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:50:11.384780 14272 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:50:11.385020 14812 raft_consensus.cc:2229] T 4de82d2f6bd44c24b874e54c10b86865 P e875b6bbdcf849538e3d71b22af61384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:50:11.385371 14812 raft_consensus.cc:2258] T 4de82d2f6bd44c24b874e54c10b86865 P e875b6bbdcf849538e3d71b22af61384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:50:11.386071 14813 ts_tablet_manager.cc:1905] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:50:11.387043 14812 ts_tablet_manager.cc:1905] T 4de82d2f6bd44c24b874e54c10b86865 P e875b6bbdcf849538e3d71b22af61384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:50:11.389503 14814 ts_tablet_manager.cc:1905] T 4de82d2f6bd44c24b874e54c10b86865 P a29624e230b7459a9ca2012108f5a54a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:50:11.390367 14812 ts_tablet_manager.cc:1918] T 4de82d2f6bd44c24b874e54c10b86865 P e875b6bbdcf849538e3d71b22af61384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:50:11.390527 14812 log.cc:1196] T 4de82d2f6bd44c24b874e54c10b86865 P e875b6bbdcf849538e3d71b22af61384: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/4de82d2f6bd44c24b874e54c10b86865
I20260928 23:50:11.390976 14812 ts_tablet_manager.cc:1939] T 4de82d2f6bd44c24b874e54c10b86865 P e875b6bbdcf849538e3d71b22af61384: Deleting consensus metadata
I20260928 23:50:11.391608 14414 tablet_service.cc:1467] Processing CreateTablet for tablet 7521a6adbb454d60ad441dcd376477bb (DEFAULT_TABLE table=test-table [id=0484dfe068ca410688b1f681a4365424]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:50:11.391898 14414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7521a6adbb454d60ad441dcd376477bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:11.392642 14554 tablet_service.cc:1467] Processing CreateTablet for tablet 7521a6adbb454d60ad441dcd376477bb (DEFAULT_TABLE table=test-table [id=0484dfe068ca410688b1f681a4365424]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:50:11.392948 14554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7521a6adbb454d60ad441dcd376477bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:11.392937 14813 ts_tablet_manager.cc:1918] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:50:11.393041 14813 log.cc:1196] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-0/wal/wals/4de82d2f6bd44c24b874e54c10b86865
I20260928 23:50:11.393394 14813 ts_tablet_manager.cc:1939] T 4de82d2f6bd44c24b874e54c10b86865 P 38f20112572b4d3591be33a57fefca51: Deleting consensus metadata
I20260928 23:50:11.393499 14814 ts_tablet_manager.cc:1918] T 4de82d2f6bd44c24b874e54c10b86865 P a29624e230b7459a9ca2012108f5a54a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:50:11.393616 14814 log.cc:1196] T 4de82d2f6bd44c24b874e54c10b86865 P a29624e230b7459a9ca2012108f5a54a: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-1/wal/wals/4de82d2f6bd44c24b874e54c10b86865
I20260928 23:50:11.394934 14814 ts_tablet_manager.cc:1939] T 4de82d2f6bd44c24b874e54c10b86865 P a29624e230b7459a9ca2012108f5a54a: Deleting consensus metadata
I20260928 23:50:11.395471 14789 tablet_bootstrap.cc:492] T 7521a6adbb454d60ad441dcd376477bb P a29624e230b7459a9ca2012108f5a54a: Bootstrap starting.
I20260928 23:50:11.396615 14789 tablet_bootstrap.cc:654] T 7521a6adbb454d60ad441dcd376477bb P a29624e230b7459a9ca2012108f5a54a: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:11.397818 14261 catalog_manager.cc:4841] TS e875b6bbdcf849538e3d71b22af61384 (127.6.152.3:46097): tablet 4de82d2f6bd44c24b874e54c10b86865 (table test-table [id=f889432c093742878079c6d6d8a89d75]) successfully deleted
I20260928 23:50:11.398497 14694 tablet_service.cc:1467] Processing CreateTablet for tablet 7521a6adbb454d60ad441dcd376477bb (DEFAULT_TABLE table=test-table [id=0484dfe068ca410688b1f681a4365424]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:50:11.398778 14694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7521a6adbb454d60ad441dcd376477bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:11.400988 14264 catalog_manager.cc:4841] TS a29624e230b7459a9ca2012108f5a54a (127.6.152.2:38143): tablet 4de82d2f6bd44c24b874e54c10b86865 (table test-table [id=f889432c093742878079c6d6d8a89d75]) successfully deleted
I20260928 23:50:11.401877 14790 tablet_bootstrap.cc:492] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384: Bootstrap starting.
I20260928 23:50:11.402838 14790 tablet_bootstrap.cc:654] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:11.403461 14264 catalog_manager.cc:4841] TS 38f20112572b4d3591be33a57fefca51 (127.6.152.1:34437): tablet 4de82d2f6bd44c24b874e54c10b86865 (table test-table [id=f889432c093742878079c6d6d8a89d75]) successfully deleted
I20260928 23:50:11.403661 14788 tablet_bootstrap.cc:492] T 7521a6adbb454d60ad441dcd376477bb P 38f20112572b4d3591be33a57fefca51: Bootstrap starting.
I20260928 23:50:11.403764 14789 tablet_bootstrap.cc:492] T 7521a6adbb454d60ad441dcd376477bb P a29624e230b7459a9ca2012108f5a54a: No bootstrap required, opened a new log
I20260928 23:50:11.403843 14789 ts_tablet_manager.cc:1397] T 7521a6adbb454d60ad441dcd376477bb P a29624e230b7459a9ca2012108f5a54a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260928 23:50:11.404361 14789 raft_consensus.cc:348] T 7521a6adbb454d60ad441dcd376477bb P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:11.404477 14789 raft_consensus.cc:374] T 7521a6adbb454d60ad441dcd376477bb P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:11.404520 14789 raft_consensus.cc:729] T 7521a6adbb454d60ad441dcd376477bb P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29624e230b7459a9ca2012108f5a54a, State: Initialized, Role: FOLLOWER
I20260928 23:50:11.404690 14788 tablet_bootstrap.cc:654] T 7521a6adbb454d60ad441dcd376477bb P 38f20112572b4d3591be33a57fefca51: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:11.404664 14789 consensus_queue.cc:260] T 7521a6adbb454d60ad441dcd376477bb P a29624e230b7459a9ca2012108f5a54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:11.404991 14789 ts_tablet_manager.cc:1428] T 7521a6adbb454d60ad441dcd376477bb P a29624e230b7459a9ca2012108f5a54a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 23:50:11.406095 14788 tablet_bootstrap.cc:492] T 7521a6adbb454d60ad441dcd376477bb P 38f20112572b4d3591be33a57fefca51: No bootstrap required, opened a new log
I20260928 23:50:11.406193 14788 ts_tablet_manager.cc:1397] T 7521a6adbb454d60ad441dcd376477bb P 38f20112572b4d3591be33a57fefca51: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 23:50:11.406723 14788 raft_consensus.cc:348] T 7521a6adbb454d60ad441dcd376477bb P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:11.406847 14788 raft_consensus.cc:374] T 7521a6adbb454d60ad441dcd376477bb P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:11.406894 14788 raft_consensus.cc:729] T 7521a6adbb454d60ad441dcd376477bb P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f20112572b4d3591be33a57fefca51, State: Initialized, Role: FOLLOWER
I20260928 23:50:11.407015 14788 consensus_queue.cc:260] T 7521a6adbb454d60ad441dcd376477bb P 38f20112572b4d3591be33a57fefca51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:11.407291 14788 ts_tablet_manager.cc:1428] T 7521a6adbb454d60ad441dcd376477bb P 38f20112572b4d3591be33a57fefca51: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 23:50:11.408185 14790 tablet_bootstrap.cc:492] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384: No bootstrap required, opened a new log
I20260928 23:50:11.408313 14790 ts_tablet_manager.cc:1397] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 23:50:11.408890 14790 raft_consensus.cc:348] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:11.409020 14790 raft_consensus.cc:374] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:11.409060 14790 raft_consensus.cc:729] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e875b6bbdcf849538e3d71b22af61384, State: Initialized, Role: FOLLOWER
I20260928 23:50:11.409199 14790 consensus_queue.cc:260] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:11.409479 14790 ts_tablet_manager.cc:1428] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:50:11.616634 14796 raft_consensus.cc:482] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:50:11.616946 14796 raft_consensus.cc:504] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:11.618213 14796 leader_election.cc:288] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38f20112572b4d3591be33a57fefca51 (127.6.152.1:34437), a29624e230b7459a9ca2012108f5a54a (127.6.152.2:38143)
I20260928 23:50:11.623696 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.619042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:55087 (local address 127.6.152.2:38143)
0928 23:50:11.619196 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:11.619202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:11.619219 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:50:11.619375 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:11.619378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:11.619418 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:11.619483 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:11.619732 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.620435 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.622266 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.622881 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.622990 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.623056 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:11.623482 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 23:50:11.623556 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 23:50:11.623696 14783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.618626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:37733 (local address 127.6.152.1:34437)
0928 23:50:11.618771 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:11.618779 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:11.618852 (+ 73us) server_negotiation.cc:407] Connection header received
0928 23:50:11.619097 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:11.619103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:11.619158 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:11.619250 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:11.619583 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.620393 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.622266 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.622880 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.622990 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.623058 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:11.623482 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 23:50:11.623556 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:50:11.623761 14818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.618488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:34437 (local address 127.6.152.3:37733)
0928 23:50:11.618768 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:11.618784 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:11.618956 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:11.619270 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:11.619272 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:11.619290 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:11.619440 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.619446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.620535 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.620539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.622126 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.622137 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.623019 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.623023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.623260 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.623385 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 23:50:11.623452 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 23:50:11.623564 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":223,"thread_start_us":91,"threads_started":1}
I20260928 23:50:11.623744 14819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.618929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:38143 (local address 127.6.152.3:55087)
0928 23:50:11.619158 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:11.619178 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:11.619262 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:11.619517 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:11.619522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:11.619545 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:11.619637 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.619641 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.620535 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.620539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.622126 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.622137 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.623019 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.623023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.623256 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.623369 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 23:50:11.623452 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 23:50:11.623556 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":268,"thread_start_us":56,"threads_started":1}
I20260928 23:50:11.624342 14574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7521a6adbb454d60ad441dcd376477bb" candidate_uuid: "e875b6bbdcf849538e3d71b22af61384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29624e230b7459a9ca2012108f5a54a" is_pre_election: true
I20260928 23:50:11.624405 14434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7521a6adbb454d60ad441dcd376477bb" candidate_uuid: "e875b6bbdcf849538e3d71b22af61384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38f20112572b4d3591be33a57fefca51" is_pre_election: true
I20260928 23:50:11.624549 14574 raft_consensus.cc:2454] T 7521a6adbb454d60ad441dcd376477bb P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e875b6bbdcf849538e3d71b22af61384 in term 0.
I20260928 23:50:11.624666 14434 raft_consensus.cc:2454] T 7521a6adbb454d60ad441dcd376477bb P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e875b6bbdcf849538e3d71b22af61384 in term 0.
I20260928 23:50:11.624962 14648 leader_election.cc:302] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a29624e230b7459a9ca2012108f5a54a, e875b6bbdcf849538e3d71b22af61384; no voters:
I20260928 23:50:11.625263 14796 raft_consensus.cc:2781] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:50:11.625339 14796 raft_consensus.cc:482] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:50:11.625391 14796 raft_consensus.cc:3037] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:11.626420 14796 raft_consensus.cc:504] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:11.626765 14796 leader_election.cc:288] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384 [CANDIDATE]: Term 1 election: Requested vote from peers 38f20112572b4d3591be33a57fefca51 (127.6.152.1:34437), a29624e230b7459a9ca2012108f5a54a (127.6.152.2:38143)
I20260928 23:50:11.627118 14434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7521a6adbb454d60ad441dcd376477bb" candidate_uuid: "e875b6bbdcf849538e3d71b22af61384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38f20112572b4d3591be33a57fefca51"
I20260928 23:50:11.627151 14574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7521a6adbb454d60ad441dcd376477bb" candidate_uuid: "e875b6bbdcf849538e3d71b22af61384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29624e230b7459a9ca2012108f5a54a"
I20260928 23:50:11.627264 14434 raft_consensus.cc:3037] T 7521a6adbb454d60ad441dcd376477bb P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:11.627300 14574 raft_consensus.cc:3037] T 7521a6adbb454d60ad441dcd376477bb P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:11.628247 14434 raft_consensus.cc:2454] T 7521a6adbb454d60ad441dcd376477bb P 38f20112572b4d3591be33a57fefca51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e875b6bbdcf849538e3d71b22af61384 in term 1.
I20260928 23:50:11.628249 14574 raft_consensus.cc:2454] T 7521a6adbb454d60ad441dcd376477bb P a29624e230b7459a9ca2012108f5a54a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e875b6bbdcf849538e3d71b22af61384 in term 1.
I20260928 23:50:11.628559 14648 leader_election.cc:302] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38f20112572b4d3591be33a57fefca51, e875b6bbdcf849538e3d71b22af61384; no voters:
I20260928 23:50:11.628734 14796 raft_consensus.cc:2781] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:50:11.628960 14796 raft_consensus.cc:686] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384 [term 1 LEADER]: Becoming Leader. State: Replica: e875b6bbdcf849538e3d71b22af61384, State: Running, Role: LEADER
I20260928 23:50:11.629225 14796 consensus_queue.cc:237] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
W20260928 23:50:11.631479 14621 tablet.cc:2369] T 7521a6adbb454d60ad441dcd376477bb P a29624e230b7459a9ca2012108f5a54a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:50:11.631484 14272 catalog_manager.cc:5495] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384 reported cstate change: term changed from 0 to 1, leader changed from <none> to e875b6bbdcf849538e3d71b22af61384 (127.6.152.3). New cstate: current_term: 1 leader_uuid: "e875b6bbdcf849538e3d71b22af61384" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } health_report { overall_health: HEALTHY } } }
W20260928 23:50:11.655524 14481 tablet.cc:2369] T 7521a6adbb454d60ad441dcd376477bb P 38f20112572b4d3591be33a57fefca51: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 23:50:11 dist-test-slave-p5bw krb5kdc[14232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639410, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.62@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM
I20260928 23:50:11.671911 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.662731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:41033 (local address 127.0.0.1:40656)
0928 23:50:11.663166 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:11.663178 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:11.663281 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:11.663601 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:11.663604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:11.664066 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:11.664249 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.664257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.665397 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.665400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.665973 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.665981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.666553 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.666555 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.666617 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.667269 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:11.667281 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:11.668763 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:11.670467 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:11.670475 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:11.670480 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:11.670715 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:11.670985 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:11.670988 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:11.670989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:11.671078 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:11.671449 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:11.671454 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:11.671556 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 23:50:11.671645 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 23:50:11.671769 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":75,"threads_started":1}
I20260928 23:50:11.671955 14827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.662794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40656 (local address 127.6.152.62:41033)
0928 23:50:11.663057 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:11.663064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:11.663258 (+ 194us) server_negotiation.cc:407] Connection header received
0928 23:50:11.663400 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:11.663403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:11.663451 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:11.663543 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:11.664362 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.665284 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.666092 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.666436 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.666528 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.669011 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:11.669014 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:11.669016 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:11.669036 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:11.670365 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:11.670799 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:11.670801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:11.670803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:11.670873 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:11.671180 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:11.671183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:11.671184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:11.671345 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:11.671417 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:11.671696 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 23:50:11.671842 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":193,"thread_start_us":109,"threads_started":1}
I20260928 23:50:11.675717 14272 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:40656:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 23:50:11.676430 14272 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:50:11.681211 14554 tablet_service.cc:1467] Processing CreateTablet for tablet 0addb746509648c6a56ca5954d62b771 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a1e940e2fa7a4b42942449078ab414bf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:50:11.681197 14414 tablet_service.cc:1467] Processing CreateTablet for tablet 0addb746509648c6a56ca5954d62b771 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a1e940e2fa7a4b42942449078ab414bf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:50:11.681586 14554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0addb746509648c6a56ca5954d62b771. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:11.681599 14414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0addb746509648c6a56ca5954d62b771. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:11.681524 14694 tablet_service.cc:1467] Processing CreateTablet for tablet 0addb746509648c6a56ca5954d62b771 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a1e940e2fa7a4b42942449078ab414bf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:50:11.681903 14694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0addb746509648c6a56ca5954d62b771. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:11.684393 14788 tablet_bootstrap.cc:492] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51: Bootstrap starting.
I20260928 23:50:11.684433 14789 tablet_bootstrap.cc:492] T 0addb746509648c6a56ca5954d62b771 P a29624e230b7459a9ca2012108f5a54a: Bootstrap starting.
I20260928 23:50:11.684743 14790 tablet_bootstrap.cc:492] T 0addb746509648c6a56ca5954d62b771 P e875b6bbdcf849538e3d71b22af61384: Bootstrap starting.
I20260928 23:50:11.685436 14789 tablet_bootstrap.cc:654] T 0addb746509648c6a56ca5954d62b771 P a29624e230b7459a9ca2012108f5a54a: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:11.685545 14788 tablet_bootstrap.cc:654] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:11.685755 14790 tablet_bootstrap.cc:654] T 0addb746509648c6a56ca5954d62b771 P e875b6bbdcf849538e3d71b22af61384: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:11.686673 14789 tablet_bootstrap.cc:492] T 0addb746509648c6a56ca5954d62b771 P a29624e230b7459a9ca2012108f5a54a: No bootstrap required, opened a new log
I20260928 23:50:11.686775 14789 ts_tablet_manager.cc:1397] T 0addb746509648c6a56ca5954d62b771 P a29624e230b7459a9ca2012108f5a54a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:50:11.686975 14790 tablet_bootstrap.cc:492] T 0addb746509648c6a56ca5954d62b771 P e875b6bbdcf849538e3d71b22af61384: No bootstrap required, opened a new log
I20260928 23:50:11.687043 14790 ts_tablet_manager.cc:1397] T 0addb746509648c6a56ca5954d62b771 P e875b6bbdcf849538e3d71b22af61384: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:50:11.687225 14789 raft_consensus.cc:348] T 0addb746509648c6a56ca5954d62b771 P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:11.687340 14789 raft_consensus.cc:374] T 0addb746509648c6a56ca5954d62b771 P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:11.687382 14789 raft_consensus.cc:729] T 0addb746509648c6a56ca5954d62b771 P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29624e230b7459a9ca2012108f5a54a, State: Initialized, Role: FOLLOWER
I20260928 23:50:11.687465 14790 raft_consensus.cc:348] T 0addb746509648c6a56ca5954d62b771 P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:11.687511 14789 consensus_queue.cc:260] T 0addb746509648c6a56ca5954d62b771 P a29624e230b7459a9ca2012108f5a54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:11.687568 14790 raft_consensus.cc:374] T 0addb746509648c6a56ca5954d62b771 P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:11.687623 14790 raft_consensus.cc:729] T 0addb746509648c6a56ca5954d62b771 P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e875b6bbdcf849538e3d71b22af61384, State: Initialized, Role: FOLLOWER
I20260928 23:50:11.687747 14790 consensus_queue.cc:260] T 0addb746509648c6a56ca5954d62b771 P e875b6bbdcf849538e3d71b22af61384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:11.687786 14789 ts_tablet_manager.cc:1428] T 0addb746509648c6a56ca5954d62b771 P a29624e230b7459a9ca2012108f5a54a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:50:11.687783 14795 tablet_replica.cc:439] T 0addb746509648c6a56ca5954d62b771 P a29624e230b7459a9ca2012108f5a54a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } } }
I20260928 23:50:11.687927 14795 tablet_replica.cc:442] T 0addb746509648c6a56ca5954d62b771 P a29624e230b7459a9ca2012108f5a54a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:50:11.688021 14790 ts_tablet_manager.cc:1428] T 0addb746509648c6a56ca5954d62b771 P e875b6bbdcf849538e3d71b22af61384: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:50:11.688216 14796 tablet_replica.cc:439] T 0addb746509648c6a56ca5954d62b771 P e875b6bbdcf849538e3d71b22af61384: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } } }
I20260928 23:50:11.688334 14796 tablet_replica.cc:442] T 0addb746509648c6a56ca5954d62b771 P e875b6bbdcf849538e3d71b22af61384: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:50:11.689594 14788 tablet_bootstrap.cc:492] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51: No bootstrap required, opened a new log
I20260928 23:50:11.689699 14788 ts_tablet_manager.cc:1397] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260928 23:50:11.690127 14788 raft_consensus.cc:348] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:11.690245 14788 raft_consensus.cc:374] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:11.690284 14788 raft_consensus.cc:729] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f20112572b4d3591be33a57fefca51, State: Initialized, Role: FOLLOWER
I20260928 23:50:11.690415 14788 consensus_queue.cc:260] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:11.690707 14788 ts_tablet_manager.cc:1428] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:50:11.690682 14809 tablet_replica.cc:439] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } } }
I20260928 23:50:11.690827 14809 tablet_replica.cc:442] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:50:11.782311 14809 raft_consensus.cc:482] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:50:11.782485 14809 raft_consensus.cc:504] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:11.783013 14809 leader_election.cc:288] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29624e230b7459a9ca2012108f5a54a (127.6.152.2:38143), e875b6bbdcf849538e3d71b22af61384 (127.6.152.3:46097)
I20260928 23:50:11.783489 14714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0addb746509648c6a56ca5954d62b771" candidate_uuid: "38f20112572b4d3591be33a57fefca51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e875b6bbdcf849538e3d71b22af61384" is_pre_election: true
I20260928 23:50:11.783489 14574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0addb746509648c6a56ca5954d62b771" candidate_uuid: "38f20112572b4d3591be33a57fefca51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29624e230b7459a9ca2012108f5a54a" is_pre_election: true
I20260928 23:50:11.783660 14574 raft_consensus.cc:2454] T 0addb746509648c6a56ca5954d62b771 P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38f20112572b4d3591be33a57fefca51 in term 0.
I20260928 23:50:11.783660 14714 raft_consensus.cc:2454] T 0addb746509648c6a56ca5954d62b771 P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38f20112572b4d3591be33a57fefca51 in term 0.
I20260928 23:50:11.784006 14365 leader_election.cc:302] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 38f20112572b4d3591be33a57fefca51, a29624e230b7459a9ca2012108f5a54a, e875b6bbdcf849538e3d71b22af61384; no voters:
I20260928 23:50:11.784221 14809 raft_consensus.cc:2781] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:50:11.784287 14809 raft_consensus.cc:482] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:50:11.784344 14809 raft_consensus.cc:3037] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:11.785517 14809 raft_consensus.cc:504] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:11.785914 14809 leader_election.cc:288] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51 [CANDIDATE]: Term 1 election: Requested vote from peers a29624e230b7459a9ca2012108f5a54a (127.6.152.2:38143), e875b6bbdcf849538e3d71b22af61384 (127.6.152.3:46097)
I20260928 23:50:11.786208 14574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0addb746509648c6a56ca5954d62b771" candidate_uuid: "38f20112572b4d3591be33a57fefca51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29624e230b7459a9ca2012108f5a54a"
I20260928 23:50:11.786340 14574 raft_consensus.cc:3037] T 0addb746509648c6a56ca5954d62b771 P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:11.786343 14714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0addb746509648c6a56ca5954d62b771" candidate_uuid: "38f20112572b4d3591be33a57fefca51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e875b6bbdcf849538e3d71b22af61384"
I20260928 23:50:11.786499 14714 raft_consensus.cc:3037] T 0addb746509648c6a56ca5954d62b771 P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:11.787322 14574 raft_consensus.cc:2454] T 0addb746509648c6a56ca5954d62b771 P a29624e230b7459a9ca2012108f5a54a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38f20112572b4d3591be33a57fefca51 in term 1.
I20260928 23:50:11.787487 14714 raft_consensus.cc:2454] T 0addb746509648c6a56ca5954d62b771 P e875b6bbdcf849538e3d71b22af61384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38f20112572b4d3591be33a57fefca51 in term 1.
I20260928 23:50:11.787810 14365 leader_election.cc:302] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38f20112572b4d3591be33a57fefca51, e875b6bbdcf849538e3d71b22af61384; no voters:
I20260928 23:50:11.788018 14809 raft_consensus.cc:2781] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:50:11.788130 14809 raft_consensus.cc:686] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51 [term 1 LEADER]: Becoming Leader. State: Replica: 38f20112572b4d3591be33a57fefca51, State: Running, Role: LEADER
I20260928 23:50:11.788262 14809 consensus_queue.cc:237] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:11.788939 14799 tablet_replica.cc:439] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51: TxnStatusTablet state changed. Reason: New leader 38f20112572b4d3591be33a57fefca51. Latest consensus state: current_term: 1 leader_uuid: "38f20112572b4d3591be33a57fefca51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } } }
I20260928 23:50:11.789057 14799 tablet_replica.cc:442] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:50:11.789345 14833 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 23:50:11.789505 14833 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 23:50:11.790004 14272 catalog_manager.cc:5495] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38f20112572b4d3591be33a57fefca51 (127.6.152.1). New cstate: current_term: 1 leader_uuid: "38f20112572b4d3591be33a57fefca51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } health_report { overall_health: UNKNOWN } } }
I20260928 23:50:11.815639 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.811998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:34437 (local address 127.0.0.1:57996)
0928 23:50:11.812171 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:11.812195 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:11.812280 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:11.812554 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:11.812556 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:11.812571 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:50:11.812660 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.812664 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.813648 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.813653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.814339 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.814348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.814933 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.814936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.815007 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.815055 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:50:11.815366 (+ 311us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:50:11.815394 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 23:50:11.815453 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 23:50:11.815526 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 23:50:11.815671 14783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.812105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57996 (local address 127.6.152.1:34437)
0928 23:50:11.812224 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:11.812229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:11.812246 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:50:11.812398 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:11.812401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:11.812454 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:11.812543 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:50:11.812756 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.813539 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.814459 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.814801 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.814908 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.815199 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:50:11.815280 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:50:11.815350 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:11.815481 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 23:50:11.815540 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 23:50:11.818171 14574 raft_consensus.cc:1261] T 0addb746509648c6a56ca5954d62b771 P a29624e230b7459a9ca2012108f5a54a [term 1 FOLLOWER]: Refusing update from remote peer 38f20112572b4d3591be33a57fefca51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:50:11.818427 14714 raft_consensus.cc:1261] T 0addb746509648c6a56ca5954d62b771 P e875b6bbdcf849538e3d71b22af61384 [term 1 FOLLOWER]: Refusing update from remote peer 38f20112572b4d3591be33a57fefca51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:50:11.818598 14809 consensus_queue.cc:1035] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:50:11.818849 14799 consensus_queue.cc:1035] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:50:11.820035 14796 tablet_replica.cc:439] T 0addb746509648c6a56ca5954d62b771 P e875b6bbdcf849538e3d71b22af61384: TxnStatusTablet state changed. Reason: New leader 38f20112572b4d3591be33a57fefca51. Latest consensus state: current_term: 1 leader_uuid: "38f20112572b4d3591be33a57fefca51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } } }
I20260928 23:50:11.820150 14796 tablet_replica.cc:442] T 0addb746509648c6a56ca5954d62b771 P e875b6bbdcf849538e3d71b22af61384: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:50:11.820220 14795 tablet_replica.cc:439] T 0addb746509648c6a56ca5954d62b771 P a29624e230b7459a9ca2012108f5a54a: TxnStatusTablet state changed. Reason: New leader 38f20112572b4d3591be33a57fefca51. Latest consensus state: current_term: 1 leader_uuid: "38f20112572b4d3591be33a57fefca51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } } }
I20260928 23:50:11.820333 14795 tablet_replica.cc:442] T 0addb746509648c6a56ca5954d62b771 P a29624e230b7459a9ca2012108f5a54a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:50:11.821704 14799 tablet_replica.cc:439] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38f20112572b4d3591be33a57fefca51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } } }
I20260928 23:50:11.821806 14799 tablet_replica.cc:442] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:50:11.822110 14799 tablet_replica.cc:439] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38f20112572b4d3591be33a57fefca51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } } }
I20260928 23:50:11.822201 14799 tablet_replica.cc:442] T 0addb746509648c6a56ca5954d62b771 P 38f20112572b4d3591be33a57fefca51: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:50:11.822675 14796 tablet_replica.cc:439] T 0addb746509648c6a56ca5954d62b771 P e875b6bbdcf849538e3d71b22af61384: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "38f20112572b4d3591be33a57fefca51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } } }
I20260928 23:50:11.822762 14796 tablet_replica.cc:442] T 0addb746509648c6a56ca5954d62b771 P e875b6bbdcf849538e3d71b22af61384: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:50:11.824697 14795 tablet_replica.cc:439] T 0addb746509648c6a56ca5954d62b771 P a29624e230b7459a9ca2012108f5a54a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "38f20112572b4d3591be33a57fefca51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } } }
I20260928 23:50:11.824798 14795 tablet_replica.cc:442] T 0addb746509648c6a56ca5954d62b771 P a29624e230b7459a9ca2012108f5a54a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:50:11.831275 14838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.827050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.6.152.3:46097)
0928 23:50:11.827699 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:11.827707 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:11.827720 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:50:11.827787 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:11.827789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:11.827849 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:11.827959 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:50:11.828178 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.828994 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.829766 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.830107 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.830204 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.830407 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:50:11.830940 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:50:11.831052 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:11.831084 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 23:50:11.831133 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":81,"threads_started":1}
I20260928 23:50:11.841380 14783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.837927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:46355 (local address 127.6.152.1:34437)
0928 23:50:11.838034 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:11.838038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:11.838183 (+ 145us) server_negotiation.cc:407] Connection header received
0928 23:50:11.838260 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:11.838263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:11.838311 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:11.838407 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:50:11.838670 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.839390 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.840230 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.840589 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.840685 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.840994 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:50:11.841069 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:50:11.841139 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:11.841211 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 23:50:11.841261 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 23:50:11.841662 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.837843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:34437 (local address 127.6.152.3:46355)
0928 23:50:11.838103 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:11.838116 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:11.838194 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:11.838429 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:11.838434 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:11.838441 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:50:11.838558 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.838563 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.839494 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.839496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.840128 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.840135 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.840724 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.840726 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.840791 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.840878 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:50:11.841148 (+ 270us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:50:11.841153 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:50:11.841201 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 23:50:11.841259 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":97,"threads_started":1}
I20260928 23:50:11.848546 14574 raft_consensus.cc:1261] T 7521a6adbb454d60ad441dcd376477bb P a29624e230b7459a9ca2012108f5a54a [term 1 FOLLOWER]: Refusing update from remote peer e875b6bbdcf849538e3d71b22af61384: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:50:11.848726 14434 raft_consensus.cc:1261] T 7521a6adbb454d60ad441dcd376477bb P 38f20112572b4d3591be33a57fefca51 [term 1 FOLLOWER]: Refusing update from remote peer e875b6bbdcf849538e3d71b22af61384: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:50:11.848999 14820 consensus_queue.cc:1035] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:50:11.849195 14796 consensus_queue.cc:1035] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:50:11.871183 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.867443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:46097 (local address 127.6.152.1:33221)
0928 23:50:11.867832 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:11.867847 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:11.867946 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:11.868179 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:11.868182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:11.868191 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:50:11.868303 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.868310 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.869320 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.869322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.869932 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:11.869959 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.870414 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.870418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:11.870501 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.870553 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:50:11.870872 (+ 319us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:50:11.870880 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 23:50:11.870952 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 23:50:11.871032 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":112,"threads_started":1}
I20260928 23:50:11.871181 14838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.867487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:33221 (local address 127.6.152.3:46097)
0928 23:50:11.867602 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:11.867607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:11.867922 (+ 315us) server_negotiation.cc:407] Connection header received
0928 23:50:11.868030 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:11.868033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:11.868092 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:11.868168 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:50:11.868410 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.869219 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.870078 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.870307 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.870416 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.870701 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:50:11.870781 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:50:11.870852 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:11.870985 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 23:50:11.871040 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 23:50:11.901309 14272 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:50:11.901495 14272 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:50:11.903724 14272 catalog_manager.cc:5782] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa: Sending DeleteTablet for 3 replicas of tablet 7521a6adbb454d60ad441dcd376477bb
I20260928 23:50:11.904345 14414 tablet_service.cc:1514] Processing DeleteTablet for tablet 7521a6adbb454d60ad441dcd376477bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:50:11 UTC) from {username='oryx', principal='oryx/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:60558
I20260928 23:50:11.904531 14694 tablet_service.cc:1514] Processing DeleteTablet for tablet 7521a6adbb454d60ad441dcd376477bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:50:11 UTC) from {username='oryx', principal='oryx/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:42096
I20260928 23:50:11.904917 14554 tablet_service.cc:1514] Processing DeleteTablet for tablet 7521a6adbb454d60ad441dcd376477bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:50:11 UTC) from {username='oryx', principal='oryx/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:46704
I20260928 23:50:11.905267 14849 tablet_replica.cc:330] T 7521a6adbb454d60ad441dcd376477bb P 38f20112572b4d3591be33a57fefca51: stopping tablet replica
I20260928 23:50:11.905411 14849 raft_consensus.cc:2229] T 7521a6adbb454d60ad441dcd376477bb P 38f20112572b4d3591be33a57fefca51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:50:11.905570 14849 raft_consensus.cc:2258] T 7521a6adbb454d60ad441dcd376477bb P 38f20112572b4d3591be33a57fefca51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:50:11.906715 14849 ts_tablet_manager.cc:1905] T 7521a6adbb454d60ad441dcd376477bb P 38f20112572b4d3591be33a57fefca51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:50:11.908955 14851 tablet_replica.cc:330] T 7521a6adbb454d60ad441dcd376477bb P a29624e230b7459a9ca2012108f5a54a: stopping tablet replica
I20260928 23:50:11.909097 14851 raft_consensus.cc:2229] T 7521a6adbb454d60ad441dcd376477bb P a29624e230b7459a9ca2012108f5a54a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:50:11.909222 14851 raft_consensus.cc:2258] T 7521a6adbb454d60ad441dcd376477bb P a29624e230b7459a9ca2012108f5a54a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:50:11.910297 14849 ts_tablet_manager.cc:1918] T 7521a6adbb454d60ad441dcd376477bb P 38f20112572b4d3591be33a57fefca51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:50:11.910398 14849 log.cc:1196] T 7521a6adbb454d60ad441dcd376477bb P 38f20112572b4d3591be33a57fefca51: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-0/wal/wals/7521a6adbb454d60ad441dcd376477bb
I20260928 23:50:11.910673 14849 ts_tablet_manager.cc:1939] T 7521a6adbb454d60ad441dcd376477bb P 38f20112572b4d3591be33a57fefca51: Deleting consensus metadata
I20260928 23:50:11.911345 14264 catalog_manager.cc:4841] TS 38f20112572b4d3591be33a57fefca51 (127.6.152.1:34437): tablet 7521a6adbb454d60ad441dcd376477bb (table test-table [id=0484dfe068ca410688b1f681a4365424]) successfully deleted
I20260928 23:50:11.913306 14851 ts_tablet_manager.cc:1905] T 7521a6adbb454d60ad441dcd376477bb P a29624e230b7459a9ca2012108f5a54a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:50:11.916033 14850 tablet_replica.cc:330] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384: stopping tablet replica
I20260928 23:50:11.916141 14851 ts_tablet_manager.cc:1918] T 7521a6adbb454d60ad441dcd376477bb P a29624e230b7459a9ca2012108f5a54a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:50:11.916168 14850 raft_consensus.cc:2229] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:50:11.916224 14851 log.cc:1196] T 7521a6adbb454d60ad441dcd376477bb P a29624e230b7459a9ca2012108f5a54a: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-1/wal/wals/7521a6adbb454d60ad441dcd376477bb
I20260928 23:50:11.916508 14851 ts_tablet_manager.cc:1939] T 7521a6adbb454d60ad441dcd376477bb P a29624e230b7459a9ca2012108f5a54a: Deleting consensus metadata
I20260928 23:50:11.916579 14850 raft_consensus.cc:2258] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:50:11.917225 14264 catalog_manager.cc:4841] TS a29624e230b7459a9ca2012108f5a54a (127.6.152.2:38143): tablet 7521a6adbb454d60ad441dcd376477bb (table test-table [id=0484dfe068ca410688b1f681a4365424]) successfully deleted
I20260928 23:50:11.917380 14850 ts_tablet_manager.cc:1905] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:50:11.920275 14850 ts_tablet_manager.cc:1918] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:50:11.920369 14850 log.cc:1196] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/7521a6adbb454d60ad441dcd376477bb
I20260928 23:50:11.920660 14850 ts_tablet_manager.cc:1939] T 7521a6adbb454d60ad441dcd376477bb P e875b6bbdcf849538e3d71b22af61384: Deleting consensus metadata
I20260928 23:50:11.921448 14261 catalog_manager.cc:4841] TS e875b6bbdcf849538e3d71b22af61384 (127.6.152.3:46097): tablet 7521a6adbb454d60ad441dcd376477bb (table test-table [id=0484dfe068ca410688b1f681a4365424]) successfully deleted
Sep 28 23:50:11 dist-test-slave-p5bw krb5kdc[14232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 23:50:11 dist-test-slave-p5bw krb5kdc[14232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:11 dist-test-slave-p5bw krb5kdc[14232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 23:50:11.941448 14827 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:11.934079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40658 (local address 127.6.152.62:41033)
0928 23:50:11.934228 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:11.934233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:11.934515 (+ 282us) server_negotiation.cc:407] Connection header received
0928 23:50:11.934696 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:11.934700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:11.934764 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:11.934863 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:11.935654 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.936255 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.937318 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.937598 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.937724 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.941259 (+ 3535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40658: BlockingRecv error: recv got EOF from 127.0.0.1:40658 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 23:50:11 dist-test-slave-p5bw krb5kdc[14232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM
I20260928 23:50:11.953292 14827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:11.943358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40664 (local address 127.6.152.62:41033)
0928 23:50:11.943497 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:11.943502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:11.944062 (+ 560us) server_negotiation.cc:407] Connection header received
0928 23:50:11.944243 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:11.944251 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:11.944301 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:11.944398 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:11.945215 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.945787 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.946674 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:11.946926 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:11.947048 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:11.949709 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:11.949713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:11.949715 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:11.949742 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:11.951346 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:11.951825 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:11.951829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:11.951831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:11.951939 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:11.952259 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:11.952264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:11.952267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:11.952485 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:11.952641 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:11.952977 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 23:50:11.953110 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 23:50:11.956364 14272 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:50:11.956990 14272 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:50:11.962355 14414 tablet_service.cc:1467] Processing CreateTablet for tablet 837f40314358498b8e7c8741e6fd0919 (DEFAULT_TABLE table=test-table [id=52389a2d2219486fb6036694ce83914b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:50:11.962615 14694 tablet_service.cc:1467] Processing CreateTablet for tablet 837f40314358498b8e7c8741e6fd0919 (DEFAULT_TABLE table=test-table [id=52389a2d2219486fb6036694ce83914b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:50:11.962697 14414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 837f40314358498b8e7c8741e6fd0919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:11.962894 14694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 837f40314358498b8e7c8741e6fd0919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:11.965495 14788 tablet_bootstrap.cc:492] T 837f40314358498b8e7c8741e6fd0919 P 38f20112572b4d3591be33a57fefca51: Bootstrap starting.
I20260928 23:50:11.965551 14790 tablet_bootstrap.cc:492] T 837f40314358498b8e7c8741e6fd0919 P e875b6bbdcf849538e3d71b22af61384: Bootstrap starting.
I20260928 23:50:11.966332 14788 tablet_bootstrap.cc:654] T 837f40314358498b8e7c8741e6fd0919 P 38f20112572b4d3591be33a57fefca51: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:11.966593 14790 tablet_bootstrap.cc:654] T 837f40314358498b8e7c8741e6fd0919 P e875b6bbdcf849538e3d71b22af61384: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:11.967932 14554 tablet_service.cc:1467] Processing CreateTablet for tablet 837f40314358498b8e7c8741e6fd0919 (DEFAULT_TABLE table=test-table [id=52389a2d2219486fb6036694ce83914b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:50:11.968246 14554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 837f40314358498b8e7c8741e6fd0919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:11.968510 14790 tablet_bootstrap.cc:492] T 837f40314358498b8e7c8741e6fd0919 P e875b6bbdcf849538e3d71b22af61384: No bootstrap required, opened a new log
I20260928 23:50:11.968600 14790 ts_tablet_manager.cc:1397] T 837f40314358498b8e7c8741e6fd0919 P e875b6bbdcf849538e3d71b22af61384: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:50:11.969094 14790 raft_consensus.cc:348] T 837f40314358498b8e7c8741e6fd0919 P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:11.969213 14790 raft_consensus.cc:374] T 837f40314358498b8e7c8741e6fd0919 P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:11.969245 14790 raft_consensus.cc:729] T 837f40314358498b8e7c8741e6fd0919 P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e875b6bbdcf849538e3d71b22af61384, State: Initialized, Role: FOLLOWER
I20260928 23:50:11.969364 14790 consensus_queue.cc:260] T 837f40314358498b8e7c8741e6fd0919 P e875b6bbdcf849538e3d71b22af61384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:11.970428 14790 ts_tablet_manager.cc:1428] T 837f40314358498b8e7c8741e6fd0919 P e875b6bbdcf849538e3d71b22af61384: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:50:11.971087 14789 tablet_bootstrap.cc:492] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a: Bootstrap starting.
I20260928 23:50:11.972162 14789 tablet_bootstrap.cc:654] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:11.974597 14789 tablet_bootstrap.cc:492] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a: No bootstrap required, opened a new log
I20260928 23:50:11.974714 14789 ts_tablet_manager.cc:1397] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 23:50:11.975415 14789 raft_consensus.cc:348] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:11.975544 14789 raft_consensus.cc:374] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:11.975589 14789 raft_consensus.cc:729] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29624e230b7459a9ca2012108f5a54a, State: Initialized, Role: FOLLOWER
I20260928 23:50:11.975742 14789 consensus_queue.cc:260] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:11.977627 14789 ts_tablet_manager.cc:1428] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:50:11.977646 14788 tablet_bootstrap.cc:492] T 837f40314358498b8e7c8741e6fd0919 P 38f20112572b4d3591be33a57fefca51: No bootstrap required, opened a new log
I20260928 23:50:11.977751 14788 ts_tablet_manager.cc:1397] T 837f40314358498b8e7c8741e6fd0919 P 38f20112572b4d3591be33a57fefca51: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260928 23:50:11.978235 14788 raft_consensus.cc:348] T 837f40314358498b8e7c8741e6fd0919 P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:11.978358 14788 raft_consensus.cc:374] T 837f40314358498b8e7c8741e6fd0919 P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:11.978398 14788 raft_consensus.cc:729] T 837f40314358498b8e7c8741e6fd0919 P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f20112572b4d3591be33a57fefca51, State: Initialized, Role: FOLLOWER
I20260928 23:50:11.978549 14788 consensus_queue.cc:260] T 837f40314358498b8e7c8741e6fd0919 P 38f20112572b4d3591be33a57fefca51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:11.978866 14788 ts_tablet_manager.cc:1428] T 837f40314358498b8e7c8741e6fd0919 P 38f20112572b4d3591be33a57fefca51: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:50:12.059626 14795 raft_consensus.cc:482] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:50:12.059777 14795 raft_consensus.cc:504] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:12.061053 14795 leader_election.cc:288] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38f20112572b4d3591be33a57fefca51 (127.6.152.1:34437), e875b6bbdcf849538e3d71b22af61384 (127.6.152.3:46097)
I20260928 23:50:12.065925 14838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:12.061463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:47401 (local address 127.6.152.3:46097)
0928 23:50:12.061618 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:12.061622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:12.062102 (+ 480us) server_negotiation.cc:407] Connection header received
0928 23:50:12.062290 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:12.062293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:12.062363 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:12.062436 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:12.062717 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:12.063464 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:12.064837 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:12.065339 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:12.065473 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:12.065537 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:12.065734 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 23:50:12.065802 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:50:12.065958 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:12.061275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:46097 (local address 127.6.152.2:47401)
0928 23:50:12.062009 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:12.062031 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:12.062138 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:12.062459 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:12.062465 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:12.062487 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:12.062610 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:12.062617 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:12.063585 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:12.063588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:12.064708 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:12.064715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:12.065488 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:12.065492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:12.065599 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:12.065632 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 23:50:12.065720 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 23:50:12.065794 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":377,"thread_start_us":70,"threads_started":1}
I20260928 23:50:12.066069 14783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:12.061455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:46577 (local address 127.6.152.1:34437)
0928 23:50:12.061579 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:12.061584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:12.061817 (+ 233us) server_negotiation.cc:407] Connection header received
0928 23:50:12.061972 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:12.061975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:12.062024 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:12.062114 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:12.062495 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:12.063129 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:12.064663 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:12.065358 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:12.065461 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:12.065526 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:12.065855 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 23:50:12.065941 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:50:12.066387 14714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837f40314358498b8e7c8741e6fd0919" candidate_uuid: "a29624e230b7459a9ca2012108f5a54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e875b6bbdcf849538e3d71b22af61384" is_pre_election: true
I20260928 23:50:12.066537 14714 raft_consensus.cc:2454] T 837f40314358498b8e7c8741e6fd0919 P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29624e230b7459a9ca2012108f5a54a in term 0.
I20260928 23:50:12.066651 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:12.061275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:34437 (local address 127.6.152.2:46577)
0928 23:50:12.061711 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:12.061728 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:12.061851 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:12.062174 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:12.062180 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:12.062199 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:12.062358 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:12.062366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:12.063260 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:12.063262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:12.064525 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:12.064536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:12.065592 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:12.065596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:12.065711 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:12.065736 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 23:50:12.066419 (+ 683us) client_negotiation.cc:240] Negotiation successful
0928 23:50:12.066519 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":89,"threads_started":1}
I20260928 23:50:12.067066 14505 leader_election.cc:302] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a29624e230b7459a9ca2012108f5a54a, e875b6bbdcf849538e3d71b22af61384; no voters:
I20260928 23:50:12.067169 14434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837f40314358498b8e7c8741e6fd0919" candidate_uuid: "a29624e230b7459a9ca2012108f5a54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38f20112572b4d3591be33a57fefca51" is_pre_election: true
I20260928 23:50:12.067301 14434 raft_consensus.cc:2454] T 837f40314358498b8e7c8741e6fd0919 P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29624e230b7459a9ca2012108f5a54a in term 0.
I20260928 23:50:12.067489 14795 raft_consensus.cc:2781] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:50:12.067552 14795 raft_consensus.cc:482] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:50:12.067616 14795 raft_consensus.cc:3037] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:12.068594 14795 raft_consensus.cc:504] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:12.069010 14795 leader_election.cc:288] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a [CANDIDATE]: Term 1 election: Requested vote from peers 38f20112572b4d3591be33a57fefca51 (127.6.152.1:34437), e875b6bbdcf849538e3d71b22af61384 (127.6.152.3:46097)
I20260928 23:50:12.069352 14434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837f40314358498b8e7c8741e6fd0919" candidate_uuid: "a29624e230b7459a9ca2012108f5a54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38f20112572b4d3591be33a57fefca51"
I20260928 23:50:12.069475 14434 raft_consensus.cc:3037] T 837f40314358498b8e7c8741e6fd0919 P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:12.069639 14714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837f40314358498b8e7c8741e6fd0919" candidate_uuid: "a29624e230b7459a9ca2012108f5a54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e875b6bbdcf849538e3d71b22af61384"
I20260928 23:50:12.069741 14714 raft_consensus.cc:3037] T 837f40314358498b8e7c8741e6fd0919 P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:12.070632 14434 raft_consensus.cc:2454] T 837f40314358498b8e7c8741e6fd0919 P 38f20112572b4d3591be33a57fefca51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29624e230b7459a9ca2012108f5a54a in term 1.
I20260928 23:50:12.070775 14714 raft_consensus.cc:2454] T 837f40314358498b8e7c8741e6fd0919 P e875b6bbdcf849538e3d71b22af61384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29624e230b7459a9ca2012108f5a54a in term 1.
I20260928 23:50:12.070998 14508 leader_election.cc:302] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38f20112572b4d3591be33a57fefca51, a29624e230b7459a9ca2012108f5a54a; no voters:
I20260928 23:50:12.071161 14795 raft_consensus.cc:2781] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:50:12.071444 14795 raft_consensus.cc:686] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a [term 1 LEADER]: Becoming Leader. State: Replica: a29624e230b7459a9ca2012108f5a54a, State: Running, Role: LEADER
I20260928 23:50:12.071878 14795 consensus_queue.cc:237] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:12.074517 14272 catalog_manager.cc:5495] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a reported cstate change: term changed from 0 to 1, leader changed from <none> to a29624e230b7459a9ca2012108f5a54a (127.6.152.2). New cstate: current_term: 1 leader_uuid: "a29624e230b7459a9ca2012108f5a54a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } health_report { overall_health: UNKNOWN } } }
I20260928 23:50:12.101523 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:12.097904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53974 (local address 127.6.152.2:38143)
0928 23:50:12.098023 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:12.098027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:12.098039 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:50:12.098115 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:12.098117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:12.098168 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:12.098257 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:50:12.098543 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:12.099040 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:12.100170 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:12.100373 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:12.100445 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:12.100718 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:50:12.101164 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:50:12.101247 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:12.101349 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 23:50:12.101392 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:50:12.103983 14714 raft_consensus.cc:1261] T 837f40314358498b8e7c8741e6fd0919 P e875b6bbdcf849538e3d71b22af61384 [term 1 FOLLOWER]: Refusing update from remote peer a29624e230b7459a9ca2012108f5a54a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:50:12.104030 14434 raft_consensus.cc:1261] T 837f40314358498b8e7c8741e6fd0919 P 38f20112572b4d3591be33a57fefca51 [term 1 FOLLOWER]: Refusing update from remote peer a29624e230b7459a9ca2012108f5a54a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:50:12.104481 14869 consensus_queue.cc:1035] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:50:12.104648 14795 consensus_queue.cc:1035] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:50:12.107481 14803 mvcc.cc:205] Tried to move back new op lower bound from 7334459031973134336 to 7334459031849099264. Current Snapshot: MvccSnapshot[applied={T|T < 7334459031973134336 or (T in {7334459031973134336})}]
I20260928 23:50:12.114800 14276 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:50:12.114933 14276 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:50:12.117957 14276 catalog_manager.cc:5782] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa: Sending DeleteTablet for 3 replicas of tablet 837f40314358498b8e7c8741e6fd0919
I20260928 23:50:12.118770 14414 tablet_service.cc:1514] Processing DeleteTablet for tablet 837f40314358498b8e7c8741e6fd0919 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:50:12 UTC) from {username='oryx', principal='oryx/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:60558
I20260928 23:50:12.118954 14554 tablet_service.cc:1514] Processing DeleteTablet for tablet 837f40314358498b8e7c8741e6fd0919 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:50:12 UTC) from {username='oryx', principal='oryx/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:46704
I20260928 23:50:12.119127 14849 tablet_replica.cc:330] T 837f40314358498b8e7c8741e6fd0919 P 38f20112572b4d3591be33a57fefca51: stopping tablet replica
I20260928 23:50:12.119251 14849 raft_consensus.cc:2229] T 837f40314358498b8e7c8741e6fd0919 P 38f20112572b4d3591be33a57fefca51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:50:12.119381 14849 raft_consensus.cc:2258] T 837f40314358498b8e7c8741e6fd0919 P 38f20112572b4d3591be33a57fefca51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:50:12.119541 14851 tablet_replica.cc:330] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a: stopping tablet replica
I20260928 23:50:12.119656 14851 raft_consensus.cc:2229] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:50:12.119915 14851 raft_consensus.cc:2258] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:50:12.120504 14694 tablet_service.cc:1514] Processing DeleteTablet for tablet 837f40314358498b8e7c8741e6fd0919 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:50:12 UTC) from {username='oryx', principal='oryx/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:42096
I20260928 23:50:12.120677 14850 tablet_replica.cc:330] T 837f40314358498b8e7c8741e6fd0919 P e875b6bbdcf849538e3d71b22af61384: stopping tablet replica
I20260928 23:50:12.120791 14850 raft_consensus.cc:2229] T 837f40314358498b8e7c8741e6fd0919 P e875b6bbdcf849538e3d71b22af61384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:50:12.120929 14850 raft_consensus.cc:2258] T 837f40314358498b8e7c8741e6fd0919 P e875b6bbdcf849538e3d71b22af61384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:50:12.121699 14850 ts_tablet_manager.cc:1905] T 837f40314358498b8e7c8741e6fd0919 P e875b6bbdcf849538e3d71b22af61384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:50:12.124261 14850 ts_tablet_manager.cc:1918] T 837f40314358498b8e7c8741e6fd0919 P e875b6bbdcf849538e3d71b22af61384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:50:12.124343 14850 log.cc:1196] T 837f40314358498b8e7c8741e6fd0919 P e875b6bbdcf849538e3d71b22af61384: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/837f40314358498b8e7c8741e6fd0919
I20260928 23:50:12.124611 14850 ts_tablet_manager.cc:1939] T 837f40314358498b8e7c8741e6fd0919 P e875b6bbdcf849538e3d71b22af61384: Deleting consensus metadata
I20260928 23:50:12.125200 14851 ts_tablet_manager.cc:1905] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:50:12.125913 14261 catalog_manager.cc:4841] TS e875b6bbdcf849538e3d71b22af61384 (127.6.152.3:46097): tablet 837f40314358498b8e7c8741e6fd0919 (table test-table [id=52389a2d2219486fb6036694ce83914b]) successfully deleted
I20260928 23:50:12.126940 14849 ts_tablet_manager.cc:1905] T 837f40314358498b8e7c8741e6fd0919 P 38f20112572b4d3591be33a57fefca51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:50:12.127539 14851 ts_tablet_manager.cc:1918] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:50:12.127624 14851 log.cc:1196] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-1/wal/wals/837f40314358498b8e7c8741e6fd0919
I20260928 23:50:12.127856 14851 ts_tablet_manager.cc:1939] T 837f40314358498b8e7c8741e6fd0919 P a29624e230b7459a9ca2012108f5a54a: Deleting consensus metadata
I20260928 23:50:12.128453 14264 catalog_manager.cc:4841] TS a29624e230b7459a9ca2012108f5a54a (127.6.152.2:38143): tablet 837f40314358498b8e7c8741e6fd0919 (table test-table [id=52389a2d2219486fb6036694ce83914b]) successfully deleted
I20260928 23:50:12.129076 14849 ts_tablet_manager.cc:1918] T 837f40314358498b8e7c8741e6fd0919 P 38f20112572b4d3591be33a57fefca51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:50:12.129172 14849 log.cc:1196] T 837f40314358498b8e7c8741e6fd0919 P 38f20112572b4d3591be33a57fefca51: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-0/wal/wals/837f40314358498b8e7c8741e6fd0919
I20260928 23:50:12.129418 14849 ts_tablet_manager.cc:1939] T 837f40314358498b8e7c8741e6fd0919 P 38f20112572b4d3591be33a57fefca51: Deleting consensus metadata
I20260928 23:50:12.129290 14272 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:50:12.129752 14272 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:50:12.130004 14264 catalog_manager.cc:4841] TS 38f20112572b4d3591be33a57fefca51 (127.6.152.1:34437): tablet 837f40314358498b8e7c8741e6fd0919 (table test-table [id=52389a2d2219486fb6036694ce83914b]) successfully deleted
I20260928 23:50:12.134112 14414 tablet_service.cc:1467] Processing CreateTablet for tablet 7bc1cfcdeed34e0e986c2bf9ebf5e76b (DEFAULT_TABLE table=test-table [id=fc3ce2f84312411cb98d82d788713f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:50:12.134203 14694 tablet_service.cc:1467] Processing CreateTablet for tablet 7bc1cfcdeed34e0e986c2bf9ebf5e76b (DEFAULT_TABLE table=test-table [id=fc3ce2f84312411cb98d82d788713f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:50:12.134188 14554 tablet_service.cc:1467] Processing CreateTablet for tablet 7bc1cfcdeed34e0e986c2bf9ebf5e76b (DEFAULT_TABLE table=test-table [id=fc3ce2f84312411cb98d82d788713f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:50:12.134352 14414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bc1cfcdeed34e0e986c2bf9ebf5e76b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:12.134413 14554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bc1cfcdeed34e0e986c2bf9ebf5e76b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:12.134423 14694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bc1cfcdeed34e0e986c2bf9ebf5e76b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:12.136446 14788 tablet_bootstrap.cc:492] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51: Bootstrap starting.
I20260928 23:50:12.136525 14789 tablet_bootstrap.cc:492] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P a29624e230b7459a9ca2012108f5a54a: Bootstrap starting.
I20260928 23:50:12.136524 14790 tablet_bootstrap.cc:492] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P e875b6bbdcf849538e3d71b22af61384: Bootstrap starting.
I20260928 23:50:12.137223 14788 tablet_bootstrap.cc:654] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:12.137353 14790 tablet_bootstrap.cc:654] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P e875b6bbdcf849538e3d71b22af61384: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:12.137362 14789 tablet_bootstrap.cc:654] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P a29624e230b7459a9ca2012108f5a54a: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:12.138444 14788 tablet_bootstrap.cc:492] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51: No bootstrap required, opened a new log
I20260928 23:50:12.138520 14788 ts_tablet_manager.cc:1397] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:50:12.138924 14789 tablet_bootstrap.cc:492] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P a29624e230b7459a9ca2012108f5a54a: No bootstrap required, opened a new log
I20260928 23:50:12.138886 14788 raft_consensus.cc:348] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:12.138985 14788 raft_consensus.cc:374] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:12.139002 14789 ts_tablet_manager.cc:1397] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P a29624e230b7459a9ca2012108f5a54a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 23:50:12.139015 14788 raft_consensus.cc:729] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f20112572b4d3591be33a57fefca51, State: Initialized, Role: FOLLOWER
I20260928 23:50:12.139128 14788 consensus_queue.cc:260] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:12.139191 14790 tablet_bootstrap.cc:492] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P e875b6bbdcf849538e3d71b22af61384: No bootstrap required, opened a new log
I20260928 23:50:12.139262 14790 ts_tablet_manager.cc:1397] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P e875b6bbdcf849538e3d71b22af61384: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 23:50:12.139367 14788 ts_tablet_manager.cc:1428] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:50:12.139441 14789 raft_consensus.cc:348] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:12.139550 14789 raft_consensus.cc:374] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:12.139583 14789 raft_consensus.cc:729] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29624e230b7459a9ca2012108f5a54a, State: Initialized, Role: FOLLOWER
I20260928 23:50:12.139629 14790 raft_consensus.cc:348] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:12.139726 14790 raft_consensus.cc:374] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:12.139698 14789 consensus_queue.cc:260] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P a29624e230b7459a9ca2012108f5a54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:12.139757 14790 raft_consensus.cc:729] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e875b6bbdcf849538e3d71b22af61384, State: Initialized, Role: FOLLOWER
I20260928 23:50:12.139863 14790 consensus_queue.cc:260] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P e875b6bbdcf849538e3d71b22af61384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:12.139950 14789 ts_tablet_manager.cc:1428] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P a29624e230b7459a9ca2012108f5a54a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:50:12.140122 14790 ts_tablet_manager.cc:1428] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P e875b6bbdcf849538e3d71b22af61384: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:50:12.288249 14809 raft_consensus.cc:482] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:50:12.288419 14809 raft_consensus.cc:504] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:12.289027 14809 leader_election.cc:288] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29624e230b7459a9ca2012108f5a54a (127.6.152.2:38143), e875b6bbdcf849538e3d71b22af61384 (127.6.152.3:46097)
I20260928 23:50:12.289496 14574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bc1cfcdeed34e0e986c2bf9ebf5e76b" candidate_uuid: "38f20112572b4d3591be33a57fefca51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29624e230b7459a9ca2012108f5a54a" is_pre_election: true
I20260928 23:50:12.289491 14714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bc1cfcdeed34e0e986c2bf9ebf5e76b" candidate_uuid: "38f20112572b4d3591be33a57fefca51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e875b6bbdcf849538e3d71b22af61384" is_pre_election: true
I20260928 23:50:12.289645 14714 raft_consensus.cc:2454] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38f20112572b4d3591be33a57fefca51 in term 0.
I20260928 23:50:12.289645 14574 raft_consensus.cc:2454] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38f20112572b4d3591be33a57fefca51 in term 0.
I20260928 23:50:12.289969 14365 leader_election.cc:302] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 38f20112572b4d3591be33a57fefca51, a29624e230b7459a9ca2012108f5a54a, e875b6bbdcf849538e3d71b22af61384; no voters:
I20260928 23:50:12.290135 14809 raft_consensus.cc:2781] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:50:12.290191 14809 raft_consensus.cc:482] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:50:12.290241 14809 raft_consensus.cc:3037] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:12.291178 14809 raft_consensus.cc:504] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:12.291535 14809 leader_election.cc:288] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51 [CANDIDATE]: Term 1 election: Requested vote from peers a29624e230b7459a9ca2012108f5a54a (127.6.152.2:38143), e875b6bbdcf849538e3d71b22af61384 (127.6.152.3:46097)
I20260928 23:50:12.291823 14574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bc1cfcdeed34e0e986c2bf9ebf5e76b" candidate_uuid: "38f20112572b4d3591be33a57fefca51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29624e230b7459a9ca2012108f5a54a"
I20260928 23:50:12.291992 14574 raft_consensus.cc:3037] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P a29624e230b7459a9ca2012108f5a54a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:12.291953 14714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bc1cfcdeed34e0e986c2bf9ebf5e76b" candidate_uuid: "38f20112572b4d3591be33a57fefca51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e875b6bbdcf849538e3d71b22af61384"
I20260928 23:50:12.292091 14714 raft_consensus.cc:3037] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P e875b6bbdcf849538e3d71b22af61384 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:12.292939 14574 raft_consensus.cc:2454] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P a29624e230b7459a9ca2012108f5a54a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38f20112572b4d3591be33a57fefca51 in term 1.
I20260928 23:50:12.293182 14714 raft_consensus.cc:2454] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P e875b6bbdcf849538e3d71b22af61384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38f20112572b4d3591be33a57fefca51 in term 1.
I20260928 23:50:12.293220 14368 leader_election.cc:302] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38f20112572b4d3591be33a57fefca51, a29624e230b7459a9ca2012108f5a54a; no voters:
I20260928 23:50:12.293378 14809 raft_consensus.cc:2781] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:50:12.293483 14809 raft_consensus.cc:686] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51 [term 1 LEADER]: Becoming Leader. State: Replica: 38f20112572b4d3591be33a57fefca51, State: Running, Role: LEADER
I20260928 23:50:12.293619 14809 consensus_queue.cc:237] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } }
I20260928 23:50:12.295203 14272 catalog_manager.cc:5495] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38f20112572b4d3591be33a57fefca51 (127.6.152.1). New cstate: current_term: 1 leader_uuid: "38f20112572b4d3591be33a57fefca51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f20112572b4d3591be33a57fefca51" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 34437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 } health_report { overall_health: UNKNOWN } } }
I20260928 23:50:12.340600 14783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:12.337789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58006 (local address 127.6.152.1:34437)
0928 23:50:12.337889 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:12.337899 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:12.337949 (+ 50us) server_negotiation.cc:407] Connection header received
0928 23:50:12.338075 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:12.338077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:12.338115 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:12.338170 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:50:12.338470 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:12.338961 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:12.339772 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:12.339968 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:12.340039 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:12.340271 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:50:12.340333 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:50:12.340378 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:12.340449 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 23:50:12.340489 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 23:50:12.348577 14783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:12.346107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:35723 (local address 127.6.152.1:34437)
0928 23:50:12.346214 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:12.346217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:12.346228 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:50:12.346260 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:12.346261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:12.346296 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:12.346354 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:50:12.346562 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:12.347019 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:12.347770 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:12.347963 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:12.348036 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:12.348219 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:50:12.348274 (+ 55us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:50:12.348325 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:12.348400 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 23:50:12.348454 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":44}
I20260928 23:50:12.348592 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:12.345895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:34437 (local address 127.6.152.1:35723)
0928 23:50:12.346030 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:12.346046 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:12.346123 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:12.346373 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:12.346375 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:12.346380 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:50:12.346476 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:12.346480 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:12.347108 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:12.347110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:12.347670 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:12.347677 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:12.348050 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:12.348052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:12.348107 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:12.348129 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:50:12.348340 (+ 211us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:50:12.348344 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:50:12.348398 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 23:50:12.348468 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":15}
I20260928 23:50:12.353623 14714 raft_consensus.cc:1261] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P e875b6bbdcf849538e3d71b22af61384 [term 1 FOLLOWER]: Refusing update from remote peer 38f20112572b4d3591be33a57fefca51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:50:12.353714 14574 raft_consensus.cc:1261] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P a29624e230b7459a9ca2012108f5a54a [term 1 FOLLOWER]: Refusing update from remote peer 38f20112572b4d3591be33a57fefca51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:50:12.353971 14809 consensus_queue.cc:1035] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e875b6bbdcf849538e3d71b22af61384" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:50:12.354115 14794 consensus_queue.cc:1035] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29624e230b7459a9ca2012108f5a54a" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 38143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:50:12.383592 14277 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:50:12.383814 14277 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:50:12.385834 14277 catalog_manager.cc:5782] T 00000000000000000000000000000000 P decdf3ad9efc41159cb3e3ef18bae1fa: Sending DeleteTablet for 3 replicas of tablet 7bc1cfcdeed34e0e986c2bf9ebf5e76b
I20260928 23:50:12.386376 14414 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bc1cfcdeed34e0e986c2bf9ebf5e76b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:50:12 UTC) from {username='oryx', principal='oryx/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:60558
I20260928 23:50:12.386445 14694 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bc1cfcdeed34e0e986c2bf9ebf5e76b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:50:12 UTC) from {username='oryx', principal='oryx/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:42096
I20260928 23:50:12.386572 14849 tablet_replica.cc:330] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51: stopping tablet replica
I20260928 23:50:12.386623 14850 tablet_replica.cc:330] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P e875b6bbdcf849538e3d71b22af61384: stopping tablet replica
I20260928 23:50:12.386706 14849 raft_consensus.cc:2229] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:50:12.386758 14850 raft_consensus.cc:2229] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P e875b6bbdcf849538e3d71b22af61384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:50:12.386909 14849 raft_consensus.cc:2258] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P 38f20112572b4d3591be33a57fefca51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:50:12.386940 14850 raft_consensus.cc:2258] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P e875b6bbdcf849538e3d71b22af61384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:50:12.386998 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 14349
I20260928 23:50:12.389124 14554 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bc1cfcdeed34e0e986c2bf9ebf5e76b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:50:12 UTC) from {username='oryx', principal='oryx/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:46704
I20260928 23:50:12.389338 14851 tablet_replica.cc:330] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P a29624e230b7459a9ca2012108f5a54a: stopping tablet replica
I20260928 23:50:12.389480 14851 raft_consensus.cc:2229] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P a29624e230b7459a9ca2012108f5a54a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:50:12.389617 14851 raft_consensus.cc:2258] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P a29624e230b7459a9ca2012108f5a54a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:50:12.390285 14851 ts_tablet_manager.cc:1905] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P a29624e230b7459a9ca2012108f5a54a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:50:12.392262 14851 ts_tablet_manager.cc:1918] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P a29624e230b7459a9ca2012108f5a54a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:50:12.392354 14851 log.cc:1196] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P a29624e230b7459a9ca2012108f5a54a: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-1/wal/wals/7bc1cfcdeed34e0e986c2bf9ebf5e76b
I20260928 23:50:12.392594 14851 ts_tablet_manager.cc:1939] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P a29624e230b7459a9ca2012108f5a54a: Deleting consensus metadata
I20260928 23:50:12.393297 14264 catalog_manager.cc:4841] TS a29624e230b7459a9ca2012108f5a54a (127.6.152.2:38143): tablet 7bc1cfcdeed34e0e986c2bf9ebf5e76b (table test-table [id=fc3ce2f84312411cb98d82d788713f93]) successfully deleted
W20260928 23:50:12.395131 14264 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.152.1:34437 (error 108)
I20260928 23:50:12.395339 14850 ts_tablet_manager.cc:1905] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P e875b6bbdcf849538e3d71b22af61384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:50:12.396241 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:12.395880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:34437 (local address 127.0.0.1:58020)
0928 23:50:12.396085 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:12.396154 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.1:34437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":50,"threads_started":1}
W20260928 23:50:12.396435 14264 catalog_manager.cc:4584] TS 38f20112572b4d3591be33a57fefca51 (127.6.152.1:34437): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7bc1cfcdeed34e0e986c2bf9ebf5e76b: Network error: Client connection negotiation failed: client connection to 127.6.152.1:34437: connect: Connection refused (error 111)
I20260928 23:50:12.397744 14850 ts_tablet_manager.cc:1918] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P e875b6bbdcf849538e3d71b22af61384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:50:12.397840 14850 log.cc:1196] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P e875b6bbdcf849538e3d71b22af61384: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/7bc1cfcdeed34e0e986c2bf9ebf5e76b
I20260928 23:50:12.398072 14850 ts_tablet_manager.cc:1939] T 7bc1cfcdeed34e0e986c2bf9ebf5e76b P e875b6bbdcf849538e3d71b22af61384: Deleting consensus metadata
I20260928 23:50:12.398705 14261 catalog_manager.cc:4841] TS e875b6bbdcf849538e3d71b22af61384 (127.6.152.3:46097): tablet 7bc1cfcdeed34e0e986c2bf9ebf5e76b (table test-table [id=fc3ce2f84312411cb98d82d788713f93]) successfully deleted
I20260928 23:50:12.398730 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 14489
I20260928 23:50:12.407653 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 14629
I20260928 23:50:12.414474 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 14245
2026-09-28T23:50:12Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3355 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:50:12 dist-test-slave-p5bw krb5kdc[14888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:50:12 dist-test-slave-p5bw krb5kdc[14888](info): set up 2 sockets
Sep 28 23:50:12 dist-test-slave-p5bw krb5kdc[14888](info): commencing operation
krb5kdc: starting...
W20260928 23:50:13.442106 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:50:13 dist-test-slave-p5bw krb5kdc[14888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:50:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:50:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:50:13.559548 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:39131
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:37263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:39131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:50:13.667649 14901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:13.667912 14901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:13.667953 14901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:13.671942 14901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:50:13.672006 14901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:13.672025 14901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:13.672042 14901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:50:13.672058 14901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:50:13.677399 14901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:37263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:39131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:39131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:13.678495 14901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:13.679507 14901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:13.688151 14907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:13.688151 14906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:13.688169 14909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:13.688187 14901 server_base.cc:1034] running on GCE node
I20260928 23:50:13.688975 14901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:13.689993 14901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:13.691179 14901 hybrid_clock.cc:648] HybridClock initialized: now 1790639413691168 us; error 27 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T23:50:13Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1281 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:50:13 dist-test-slave-p5bw krb5kdc[14914](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:50:13 dist-test-slave-p5bw krb5kdc[14914](info): set up 2 sockets
Sep 28 23:50:13 dist-test-slave-p5bw krb5kdc[14914](info): commencing operation
krb5kdc: starting...
W20260928 23:50:14.758345 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:50:14 dist-test-slave-p5bw krb5kdc[14914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:50:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:50:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:50:14.880237 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:40931
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:40931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:50:14.989677 14927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:14.989969 14927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:14.990047 14927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:14.994045 14927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:50:14.994120 14927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:14.994158 14927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:14.994215 14927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:50:14.994252 14927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:50:14.999457 14927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:40931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:40931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:15.000500 14927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:15.001565 14927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:15.009666 14933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:15.009742 14927 server_base.cc:1034] running on GCE node
W20260928 23:50:15.009665 14935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:15.009654 14932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:15.010500 14927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:15.011569 14927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:15.012758 14927 hybrid_clock.cc:648] HybridClock initialized: now 1790639415012745 us; error 24 us; skew 500 ppm
Sep 28 23:50:15 dist-test-slave-p5bw krb5kdc[14914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639415, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:15.027271 14927 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:50:15.028272 14927 webserver.cc:460] Webserver started at http://127.6.152.62:46299/ using document root <none> and password file <none>
I20260928 23:50:15.028914 14927 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:15.029006 14927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:15.029244 14927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:15.031338 14927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "2f325f895fa64e349d2eb5edb68e38e5"
format_stamp: "Formatted at 2026-09-28 23:50:15 on dist-test-slave-p5bw"
server_key: "29c9118dfd30e9cd181976d1febae441"
server_key_iv: "6ecbe5d8347deb1f0b162a2df894d002"
server_key_version: "encryptionkey@0"
I20260928 23:50:15.031894 14927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "2f325f895fa64e349d2eb5edb68e38e5"
format_stamp: "Formatted at 2026-09-28 23:50:15 on dist-test-slave-p5bw"
server_key: "29c9118dfd30e9cd181976d1febae441"
server_key_iv: "6ecbe5d8347deb1f0b162a2df894d002"
server_key_version: "encryptionkey@0"
I20260928 23:50:15.035502 14927 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 23:50:15.037911 14942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:15.039040 14927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:50:15.039186 14927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "2f325f895fa64e349d2eb5edb68e38e5"
format_stamp: "Formatted at 2026-09-28 23:50:15 on dist-test-slave-p5bw"
server_key: "29c9118dfd30e9cd181976d1febae441"
server_key_iv: "6ecbe5d8347deb1f0b162a2df894d002"
server_key_version: "encryptionkey@0"
I20260928 23:50:15.039292 14927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:15.056394 14927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:15.059128 14927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:15.059329 14927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:15.066299 14927 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:40931
I20260928 23:50:15.066387 14994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:40931 every 8 connection(s)
I20260928 23:50:15.067257 14927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:50:15.069959 14995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:15.075271 14995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5: Bootstrap starting.
I20260928 23:50:15.076768 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 14927
I20260928 23:50:15.076921 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:50:15.077248 6752 external_mini_cluster.cc:1428] Setting key 03e33ba7d71ac3e732335cfbd490ce6b
I20260928 23:50:15.078016 14995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:15.078891 14995 log.cc:824] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5: Log is configured to *not* fsync() on all Append() calls
I20260928 23:50:15.080739 14995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5: No bootstrap required, opened a new log
I20260928 23:50:15.084117 14995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f325f895fa64e349d2eb5edb68e38e5" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40931 } }
I20260928 23:50:15.084362 14995 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:15.084451 14995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f325f895fa64e349d2eb5edb68e38e5, State: Initialized, Role: FOLLOWER
I20260928 23:50:15.084973 14995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f325f895fa64e349d2eb5edb68e38e5" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40931 } }
I20260928 23:50:15.085124 14995 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:50:15.085193 14995 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:50:15.085289 14995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 23:50:15 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:50:15.086241 14995 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f325f895fa64e349d2eb5edb68e38e5" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40931 } }
I20260928 23:50:15.086584 14995 leader_election.cc:302] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f325f895fa64e349d2eb5edb68e38e5; no voters:
I20260928 23:50:15.086916 14995 leader_election.cc:288] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:50:15.087001 15000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:50:15.087210 15000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [term 1 LEADER]: Becoming Leader. State: Replica: 2f325f895fa64e349d2eb5edb68e38e5, State: Running, Role: LEADER
I20260928 23:50:15.087553 15000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f325f895fa64e349d2eb5edb68e38e5" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40931 } }
I20260928 23:50:15.088236 14995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:50:15.089351 15002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f325f895fa64e349d2eb5edb68e38e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f325f895fa64e349d2eb5edb68e38e5" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40931 } } }
I20260928 23:50:15.089762 15002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:15.089649 15001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f325f895fa64e349d2eb5edb68e38e5. Latest consensus state: current_term: 1 leader_uuid: "2f325f895fa64e349d2eb5edb68e38e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f325f895fa64e349d2eb5edb68e38e5" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40931 } } }
I20260928 23:50:15.090575 15001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:15.091897 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:15.078401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45338 (local address 127.6.152.62:40931)
0928 23:50:15.078815 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:15.078830 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:15.078863 (+ 33us) server_negotiation.cc:407] Connection header received
0928 23:50:15.079716 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:15.079737 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:15.080053 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:15.080385 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:15.081237 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:15.082192 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:15.083066 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:15.083536 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:15.083716 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:15.086423 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:15.086428 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:15.086441 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:15.086466 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:15.088434 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:15.088956 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:15.088961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:15.088967 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:15.089108 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:15.089417 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:15.089423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:15.089425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:15.089923 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:15.090116 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:15.090883 (+ 767us) server_negotiation.cc:299] Negotiation successful
0928 23:50:15.091190 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":137,"threads_started":1}
W20260928 23:50:15.093361 15016 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:50:15.093451 15016 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:50:15.093561 15012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:50:15.094403 15012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:50:15.099712 15012 catalog_manager.cc:1347] Generated new cluster ID: 5fd62cd116ac4492b973ead8bf9cc91d
I20260928 23:50:15.099787 15012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:50:15.126310 15012 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:50:15.127282 15012 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:50:15.135794 15012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5: Generated new TSK 0
I20260928 23:50:15.136507 15012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
I20260928 23:50:15.171732 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:40931
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:50:15.281656 15021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:15.281893 15021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:15.281934 15021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:15.285930 15021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:15.285987 15021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:15.286090 15021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:50:15.291630 15021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:40931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:15.292604 15021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:15.293632 15021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:15.302759 15029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:15.302735 15026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:15.302809 15021 server_base.cc:1034] running on GCE node
W20260928 23:50:15.302760 15027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:15.303318 15021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:15.304100 15021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:15.305310 15021 hybrid_clock.cc:648] HybridClock initialized: now 1790639415305293 us; error 24 us; skew 500 ppm
Sep 28 23:50:15 dist-test-slave-p5bw krb5kdc[14914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639415, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:15.308073 15021 init.cc:377] Logged in from keytab as kudu/127.6.152.1@KRBTEST.COM (short username kudu)
I20260928 23:50:15.308967 15021 webserver.cc:460] Webserver started at http://127.6.152.1:35269/ using document root <none> and password file <none>
I20260928 23:50:15.309507 15021 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:15.309595 15021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:15.309823 15021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:15.311786 15021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "ebda463638cc4436be1dcb6a1dcca61c"
format_stamp: "Formatted at 2026-09-28 23:50:15 on dist-test-slave-p5bw"
server_key: "e420ef4a2ef505554b619ff1be623572"
server_key_iv: "02a474967b5df4f2c590353192fa3a18"
server_key_version: "encryptionkey@0"
I20260928 23:50:15.312469 15021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "ebda463638cc4436be1dcb6a1dcca61c"
format_stamp: "Formatted at 2026-09-28 23:50:15 on dist-test-slave-p5bw"
server_key: "e420ef4a2ef505554b619ff1be623572"
server_key_iv: "02a474967b5df4f2c590353192fa3a18"
server_key_version: "encryptionkey@0"
I20260928 23:50:15.316033 15021 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260928 23:50:15.318517 15036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:15.319658 15021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 23:50:15.319774 15021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "ebda463638cc4436be1dcb6a1dcca61c"
format_stamp: "Formatted at 2026-09-28 23:50:15 on dist-test-slave-p5bw"
server_key: "e420ef4a2ef505554b619ff1be623572"
server_key_iv: "02a474967b5df4f2c590353192fa3a18"
server_key_version: "encryptionkey@0"
I20260928 23:50:15.319873 15021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:15.339752 15021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:15.342821 15021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:15.343030 15021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:15.343720 15021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:50:15.344542 15021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:15.344590 15021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:15.344659 15021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:15.344691 15021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:15.353173 15021 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:39293
I20260928 23:50:15.353196 15149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:39293 every 8 connection(s)
I20260928 23:50:15.354022 15021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:50:15.358675 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 15021
I20260928 23:50:15.358814 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
I20260928 23:50:15.359138 6752 external_mini_cluster.cc:1428] Setting key ce0ac56004df2f7f614bb5db94481f58
Sep 28 23:50:15 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639415, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:50:15.368690 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:15.355847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:46525 (local address 127.6.152.62:40931)
0928 23:50:15.355995 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:15.355999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:15.357049 (+ 1050us) server_negotiation.cc:407] Connection header received
0928 23:50:15.357896 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:15.357899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:15.357943 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:15.358016 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:15.359651 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:15.360226 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:15.361970 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:15.362224 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:15.362365 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:15.364779 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:15.364786 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:15.364791 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:15.364847 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:15.366563 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:15.367042 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:15.367049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:15.367052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:15.367154 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:15.367475 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:15.367478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:15.367482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:15.367736 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:15.367842 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:15.368322 (+ 480us) server_negotiation.cc:299] Negotiation successful
0928 23:50:15.368479 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.6.152.2@KRBTEST.COM; defaulting to no policy
I20260928 23:50:15.369470 15152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:15.356226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.1:46525)
0928 23:50:15.356925 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:15.356967 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:15.357716 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:15.358247 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:15.358254 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:15.358785 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:15.359475 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:15.359495 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:15.360392 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:15.360396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:15.361848 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:15.361863 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:15.362419 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:15.362423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:15.362592 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:15.363212 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:15.363227 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:15.364614 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:15.366677 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:15.366682 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:15.366695 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:15.366939 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:15.367247 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:15.367249 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:15.367250 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:15.367374 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:15.367845 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:15.367850 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:15.368078 (+ 228us) client_negotiation.cc:769] Sending connection context
0928 23:50:15.368231 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 23:50:15.368478 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":117,"threads_started":1}
I20260928 23:50:15.370723 15150 heartbeater.cc:347] Connected to a master server at 127.6.152.62:40931
I20260928 23:50:15.370977 15150 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:15.371472 15150 heartbeater.cc:511] Master 127.6.152.62:40931 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.2@KRBTEST.COM" created.
I20260928 23:50:15.373106 14959 ts_manager.cc:194] Registered new tserver with Master: ebda463638cc4436be1dcb6a1dcca61c (127.6.152.1:39293)
I20260928 23:50:15.374219 14959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.1@KRBTEST.COM'} at 127.6.152.1:46525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
I20260928 23:50:15.388253 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:40931
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:50:15.495558 15155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:15.495803 15155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:15.495873 15155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:15.499850 15155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:15.499900 15155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:15.499995 15155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:50:15.505081 15155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:40931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:15.506033 15155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:15.507059 15155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:15.516103 15161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:15.516124 15163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:15.516108 15160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:15.516222 15155 server_base.cc:1034] running on GCE node
I20260928 23:50:15.516709 15155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:15.517364 15155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:15.518554 15155 hybrid_clock.cc:648] HybridClock initialized: now 1790639415518542 us; error 30 us; skew 500 ppm
Sep 28 23:50:15 dist-test-slave-p5bw krb5kdc[14914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639415, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:15.521188 15155 init.cc:377] Logged in from keytab as kudu/127.6.152.2@KRBTEST.COM (short username kudu)
I20260928 23:50:15.522059 15155 webserver.cc:460] Webserver started at http://127.6.152.2:38231/ using document root <none> and password file <none>
I20260928 23:50:15.522605 15155 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:15.522663 15155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:15.522899 15155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:15.525014 15155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "6a7ae29fedfa499abc5ec21311d42db3"
format_stamp: "Formatted at 2026-09-28 23:50:15 on dist-test-slave-p5bw"
server_key: "7bacd7c223a2cc53ee3ccdbf5d46fc85"
server_key_iv: "e7395ae8b8af1976992e7dd519465bb8"
server_key_version: "encryptionkey@0"
I20260928 23:50:15.525556 15155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "6a7ae29fedfa499abc5ec21311d42db3"
format_stamp: "Formatted at 2026-09-28 23:50:15 on dist-test-slave-p5bw"
server_key: "7bacd7c223a2cc53ee3ccdbf5d46fc85"
server_key_iv: "e7395ae8b8af1976992e7dd519465bb8"
server_key_version: "encryptionkey@0"
I20260928 23:50:15.529016 15155 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:50:15.531265 15170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:15.532367 15155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:50:15.532480 15155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "6a7ae29fedfa499abc5ec21311d42db3"
format_stamp: "Formatted at 2026-09-28 23:50:15 on dist-test-slave-p5bw"
server_key: "7bacd7c223a2cc53ee3ccdbf5d46fc85"
server_key_iv: "e7395ae8b8af1976992e7dd519465bb8"
server_key_version: "encryptionkey@0"
I20260928 23:50:15.532578 15155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:15.542704 15155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:15.545411 15155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:15.545598 15155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:15.546144 15155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:50:15.546891 15155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:15.546936 15155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:15.547003 15155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:15.547049 15155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:15.555310 15155 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:37129
I20260928 23:50:15.555338 15284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:37129 every 8 connection(s)
I20260928 23:50:15.556143 15155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:50:15.563982 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 15155
I20260928 23:50:15.564122 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:50:15.564486 6752 external_mini_cluster.cc:1428] Setting key 5186fde80988e679c416e795776cd6af
Sep 28 23:50:15 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639415, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:50:15.569276 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:15.557885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:49369 (local address 127.6.152.62:40931)
0928 23:50:15.558028 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:15.558033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:15.559066 (+ 1033us) server_negotiation.cc:407] Connection header received
0928 23:50:15.559869 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:15.559871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:15.559915 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:15.559986 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:15.561135 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:15.561656 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:15.562706 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:15.562923 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:15.562996 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:15.565566 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:15.565570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:15.565580 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:15.565603 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:15.567001 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:15.567618 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:15.567623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:15.567628 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:15.567722 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:15.568066 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:15.568079 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:15.568083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:15.568321 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:15.568426 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:15.568893 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 23:50:15.569028 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:50:15.569970 15287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:15.558313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.2:49369)
0928 23:50:15.558928 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:15.558967 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:15.559699 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:15.560145 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:15.560152 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:15.560475 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:15.561014 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:15.561029 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:15.561762 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:15.561765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:15.562608 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:15.562616 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:15.563041 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:15.563044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:15.563175 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:15.563678 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:15.563693 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:15.565416 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:15.567115 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:15.567123 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:15.567141 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:15.567509 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:15.567819 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:15.567821 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:15.567822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:15.567971 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:15.568422 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:15.568427 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:15.568642 (+ 215us) client_negotiation.cc:769] Sending connection context
0928 23:50:15.568793 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 23:50:15.569099 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":108,"threads_started":1}
I20260928 23:50:15.571077 15285 heartbeater.cc:347] Connected to a master server at 127.6.152.62:40931
I20260928 23:50:15.571305 15285 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:15.571760 15285 heartbeater.cc:511] Master 127.6.152.62:40931 requested a full tablet report, sending...
I20260928 23:50:15.572745 14959 ts_manager.cc:194] Registered new tserver with Master: 6a7ae29fedfa499abc5ec21311d42db3 (127.6.152.2:37129)
I20260928 23:50:15.573323 14959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.2@KRBTEST.COM'} at 127.6.152.2:49369
WARNING: no policy specified for kudu/127.6.152.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
I20260928 23:50:15.590118 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:40931
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:50:15.697273 15290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:15.697573 15290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:15.697647 15290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:15.701704 15290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:15.701793 15290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:15.701887 15290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:50:15.707125 15290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:40931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:15.708225 15290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:15.709275 15290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:15.718715 15298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:15.718766 15295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:15.718690 15290 server_base.cc:1034] running on GCE node
W20260928 23:50:15.718715 15296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:15.719388 15290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:15.720044 15290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:15.721201 15290 hybrid_clock.cc:648] HybridClock initialized: now 1790639415721165 us; error 45 us; skew 500 ppm
Sep 28 23:50:15 dist-test-slave-p5bw krb5kdc[14914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639415, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:15.724030 15290 init.cc:377] Logged in from keytab as kudu/127.6.152.3@KRBTEST.COM (short username kudu)
I20260928 23:50:15.724898 15290 webserver.cc:460] Webserver started at http://127.6.152.3:43939/ using document root <none> and password file <none>
I20260928 23:50:15.725435 15290 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:15.725522 15290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:15.725744 15290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:15.727636 15290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "7503066d06c24264b8ca2d6bb4587f7b"
format_stamp: "Formatted at 2026-09-28 23:50:15 on dist-test-slave-p5bw"
server_key: "43449ef75853c09cdcc162c46d19561f"
server_key_iv: "a838cb14b8d8d66a3df3ccc21bc60859"
server_key_version: "encryptionkey@0"
I20260928 23:50:15.728117 15290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "7503066d06c24264b8ca2d6bb4587f7b"
format_stamp: "Formatted at 2026-09-28 23:50:15 on dist-test-slave-p5bw"
server_key: "43449ef75853c09cdcc162c46d19561f"
server_key_iv: "a838cb14b8d8d66a3df3ccc21bc60859"
server_key_version: "encryptionkey@0"
I20260928 23:50:15.731608 15290 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 23:50:15.733919 15305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:15.735085 15290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:50:15.735212 15290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "7503066d06c24264b8ca2d6bb4587f7b"
format_stamp: "Formatted at 2026-09-28 23:50:15 on dist-test-slave-p5bw"
server_key: "43449ef75853c09cdcc162c46d19561f"
server_key_iv: "a838cb14b8d8d66a3df3ccc21bc60859"
server_key_version: "encryptionkey@0"
I20260928 23:50:15.735321 15290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:15.755223 15290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:15.758234 15290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:15.758435 15290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:15.759033 15290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:50:15.759841 15290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:15.759909 15290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:15.759977 15290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:15.760028 15290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:15.768704 15290 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:36025
I20260928 23:50:15.768735 15418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:36025 every 8 connection(s)
I20260928 23:50:15.769594 15290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:50:15.776575 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 15290
I20260928 23:50:15.776711 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:50:15.777114 6752 external_mini_cluster.cc:1428] Setting key 696eb4dd7279eab6f6eb48ee47337c35
Sep 28 23:50:15 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639415, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:50:15.782343 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:15.771308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:40699 (local address 127.6.152.62:40931)
0928 23:50:15.771453 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:15.771456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:15.772451 (+ 995us) server_negotiation.cc:407] Connection header received
0928 23:50:15.773342 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:15.773345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:15.773391 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:15.773463 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:15.774531 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:15.775043 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:15.776121 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:15.776367 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:15.776475 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:15.779156 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:15.779158 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:15.779160 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:15.779182 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:15.780361 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:15.780883 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:15.780887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:15.780888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:15.780968 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:15.781303 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:15.781306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:15.781308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:15.781477 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:15.781582 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:15.782054 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 23:50:15.782180 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 23:50:15.783073 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:15.771693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.3:40699)
0928 23:50:15.772307 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:15.772346 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:15.773155 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:15.773626 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:15.773633 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:15.773961 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:15.774411 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:15.774423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:15.775141 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:15.775143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:15.776018 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:15.776027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:15.776501 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:15.776504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:15.776642 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:15.777328 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:15.777343 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:15.778904 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:15.780476 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:15.780482 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:15.780495 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:15.780749 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:15.781068 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:15.781071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:15.781072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:15.781203 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:15.781596 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:15.781604 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:15.781828 (+ 224us) client_negotiation.cc:769] Sending connection context
0928 23:50:15.781998 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 23:50:15.782300 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":107,"threads_started":1}
I20260928 23:50:15.784088 15419 heartbeater.cc:347] Connected to a master server at 127.6.152.62:40931
I20260928 23:50:15.784322 15419 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:15.784782 15419 heartbeater.cc:511] Master 127.6.152.62:40931 requested a full tablet report, sending...
I20260928 23:50:15.785768 14959 ts_manager.cc:194] Registered new tserver with Master: 7503066d06c24264b8ca2d6bb4587f7b (127.6.152.3:36025)
I20260928 23:50:15.786314 14959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.3@KRBTEST.COM'} at 127.6.152.3:40699
I20260928 23:50:15.790946 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.1.keytab.
Entry for principal oryx/127.6.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.1.keytab.
WARNING: no policy specified for oryx/127.6.152.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.152.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.152.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.2.keytab.
Entry for principal oryx/127.6.152.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.2.keytab.
WARNING: no policy specified for oryx/127.6.152.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.152.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.152.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.3.keytab.
Entry for principal oryx/127.6.152.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.3.keytab.
I20260928 23:50:15.864336 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 15021
I20260928 23:50:15.869921 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 15155
I20260928 23:50:15.875363 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 15290
I20260928 23:50:15.881626 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 14927
I20260928 23:50:15.887632 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:40931
--webserver_interface=127.6.152.62
--webserver_port=46299
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:40931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:50:15.995052 15428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:15.995307 15428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:15.995363 15428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:15.999269 15428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:50:15.999325 15428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:15.999344 15428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:15.999361 15428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:50:15.999377 15428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:50:16.004456 15428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:40931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:40931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=46299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:16.005470 15428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:16.006408 15428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:16.014835 15434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:16.014870 15433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:16.014883 15436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:16.014956 15428 server_base.cc:1034] running on GCE node
I20260928 23:50:16.015651 15428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:16.016726 15428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:16.017882 15428 hybrid_clock.cc:648] HybridClock initialized: now 1790639416017869 us; error 27 us; skew 500 ppm
Sep 28 23:50:16 dist-test-slave-p5bw krb5kdc[14914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639416, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:16.020545 15428 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:50:16.021436 15428 webserver.cc:460] Webserver started at http://127.6.152.62:46299/ using document root <none> and password file <none>
I20260928 23:50:16.021978 15428 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:16.022094 15428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:16.025887 15428 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 23:50:16.027904 15443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:16.029119 15428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:50:16.029248 15428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "2f325f895fa64e349d2eb5edb68e38e5"
format_stamp: "Formatted at 2026-09-28 23:50:15 on dist-test-slave-p5bw"
server_key: "29c9118dfd30e9cd181976d1febae441"
server_key_iv: "6ecbe5d8347deb1f0b162a2df894d002"
server_key_version: "encryptionkey@0"
I20260928 23:50:16.029771 15428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:16.048415 15428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:16.051240 15428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:16.051440 15428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:16.058387 15428 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:40931
I20260928 23:50:16.058405 15495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:40931 every 8 connection(s)
I20260928 23:50:16.059386 15428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:50:16.063412 15496 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:50:16.063832 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 15428
I20260928 23:50:16.064327 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:39293
--local_ip_for_outbound_sockets=127.6.152.1
--tserver_master_addrs=127.6.152.62:40931
--webserver_port=35269
--webserver_interface=127.6.152.1
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.152.1
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:50:16.065560 15496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5: Bootstrap starting.
I20260928 23:50:16.081995 15496 log.cc:824] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5: Log is configured to *not* fsync() on all Append() calls
I20260928 23:50:16.088415 15496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:50:16.088795 15496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5: Bootstrap complete.
I20260928 23:50:16.093042 15496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f325f895fa64e349d2eb5edb68e38e5" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40931 } }
I20260928 23:50:16.093430 15496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f325f895fa64e349d2eb5edb68e38e5, State: Initialized, Role: FOLLOWER
I20260928 23:50:16.093917 15496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f325f895fa64e349d2eb5edb68e38e5" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40931 } }
I20260928 23:50:16.094029 15496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:50:16.094116 15496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:50:16.094214 15496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:50:16.096078 15496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f325f895fa64e349d2eb5edb68e38e5" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40931 } }
I20260928 23:50:16.096418 15496 leader_election.cc:302] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f325f895fa64e349d2eb5edb68e38e5; no voters:
I20260928 23:50:16.096727 15496 leader_election.cc:288] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 23:50:16.096882 15500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 23:50:16.097119 15500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [term 2 LEADER]: Becoming Leader. State: Replica: 2f325f895fa64e349d2eb5edb68e38e5, State: Running, Role: LEADER
I20260928 23:50:16.097504 15500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f325f895fa64e349d2eb5edb68e38e5" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40931 } }
I20260928 23:50:16.097990 15496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:50:16.098876 15501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2f325f895fa64e349d2eb5edb68e38e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f325f895fa64e349d2eb5edb68e38e5" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40931 } } }
I20260928 23:50:16.098989 15501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:16.099661 15502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f325f895fa64e349d2eb5edb68e38e5. Latest consensus state: current_term: 2 leader_uuid: "2f325f895fa64e349d2eb5edb68e38e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f325f895fa64e349d2eb5edb68e38e5" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40931 } } }
I20260928 23:50:16.099836 15502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5 [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:16.100116 15509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:50:16.103796 15509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:50:16.104423 15509 catalog_manager.cc:1259] Loaded cluster ID: 5fd62cd116ac4492b973ead8bf9cc91d
I20260928 23:50:16.104481 15509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:50:16.106732 15509 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:50:16.107403 15509 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2f325f895fa64e349d2eb5edb68e38e5: Loaded TSK: 0
I20260928 23:50:16.108594 15509 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:50:16.176234 15498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:16.176463 15498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:16.176502 15498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:16.180428 15498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:16.180478 15498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:16.180567 15498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:50:16.185606 15498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.1.keytab
--principal=oryx/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:39293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=35269
--tserver_master_addrs=127.6.152.62:40931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:16.186525 15498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:16.187469 15498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:16.196501 15523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:16.196511 15525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:16.196666 15522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:16.196777 15498 server_base.cc:1034] running on GCE node
I20260928 23:50:16.197342 15498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:16.198115 15498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:16.199312 15498 hybrid_clock.cc:648] HybridClock initialized: now 1790639416199290 us; error 31 us; skew 500 ppm
Sep 28 23:50:16 dist-test-slave-p5bw krb5kdc[14914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639416, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:16.202725 15498 init.cc:377] Logged in from keytab as oryx/127.6.152.1@KRBTEST.COM (short username oryx)
I20260928 23:50:16.203787 15498 webserver.cc:460] Webserver started at http://127.6.152.1:35269/ using document root <none> and password file <none>
I20260928 23:50:16.204483 15498 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:16.204589 15498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:16.210168 15498 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:50:16.212271 15532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:16.213537 15498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:50:16.213634 15498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "ebda463638cc4436be1dcb6a1dcca61c"
format_stamp: "Formatted at 2026-09-28 23:50:15 on dist-test-slave-p5bw"
server_key: "e420ef4a2ef505554b619ff1be623572"
server_key_iv: "02a474967b5df4f2c590353192fa3a18"
server_key_version: "encryptionkey@0"
I20260928 23:50:16.214172 15498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:16.228144 15498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:16.231307 15498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:16.231511 15498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:16.232089 15498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:50:16.232939 15498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:16.232983 15498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:16.233044 15498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:16.233074 15498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:16.242906 15498 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:39293
I20260928 23:50:16.242935 15645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:39293 every 8 connection(s)
I20260928 23:50:16.243889 15498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:50:16.250582 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 15498
I20260928 23:50:16.251025 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:37129
--local_ip_for_outbound_sockets=127.6.152.2
--tserver_master_addrs=127.6.152.62:40931
--webserver_port=38231
--webserver_interface=127.6.152.2
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.152.2
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:50:16 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:16 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 23:50:16.257532 15648 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:16.246144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:36099 (local address 127.6.152.62:40931)
0928 23:50:16.246633 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:16.246658 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:16.247135 (+ 477us) server_negotiation.cc:407] Connection header received
0928 23:50:16.248232 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:16.248242 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:16.248523 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:16.248866 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:16.250013 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.251049 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.252381 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.252901 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.253023 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:16.256774 (+ 3751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.1:36099: BlockingRecv error: recv got EOF from 127.6.152.1:36099 (error 108)
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":112,"threads_started":1}
I20260928 23:50:16.257541 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:16.246277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.1:36099)
0928 23:50:16.246969 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:16.247009 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:16.247821 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:16.248949 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:16.248957 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:16.249382 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:16.249837 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:16.249850 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.251245 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.251247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:16.252233 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:16.252245 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.253078 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.253085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:16.253309 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:16.253905 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:16.253930 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:16.256582 (+ 2652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":130,"threads_started":1}
W20260928 23:50:16.258249 15646 heartbeater.cc:644] Failed to heartbeat to 127.6.152.62:40931 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.152.62:40931: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Sep 28 23:50:16 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:16 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:16.263566 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:16.258695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.1:41105)
0928 23:50:16.258803 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:16.258819 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:16.258914 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:16.259157 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:16.259160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:16.259311 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:16.259395 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:16.259399 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.260104 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.260108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:16.260645 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:16.260652 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.261144 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.261147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:16.261242 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:16.261725 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:16.261736 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:16.263429 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 23:50:16.263657 15648 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:16.258825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:41105 (local address 127.6.152.62:40931)
0928 23:50:16.258962 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:16.258967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:16.258981 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:50:16.259021 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:16.259023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:16.259058 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:16.259143 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:16.259501 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.259992 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.260760 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.261031 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.261144 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:16.263542 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.1:41105: BlockingRecv error: recv got EOF from 127.6.152.1:41105 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 23:50:16 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:16 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:16.269313 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:16.264254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.1:57009)
0928 23:50:16.264431 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:16.264441 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:16.264497 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:16.264701 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:16.264703 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:16.264862 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:16.264965 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:16.264974 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.265973 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.265975 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:16.266470 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:16.266477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.267045 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.267048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:16.267110 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:16.267551 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:16.267562 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:16.269159 (+ 1597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109}
W20260928 23:50:16.269344 15648 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:16.264355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:57009 (local address 127.6.152.62:40931)
0928 23:50:16.264476 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:16.264480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:16.264494 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:50:16.264557 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:16.264560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:16.264600 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:16.264681 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:16.265082 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.265869 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.266568 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.266939 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.267074 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:16.269266 (+ 2192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.1:57009: BlockingRecv error: recv got EOF from 127.6.152.1:57009 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 23:50:16.269502 15646 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:50:16.363883 15650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:16.364133 15650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:16.364176 15650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:16.368122 15650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:16.368184 15650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:16.368242 15650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:50:16.373355 15650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.2.keytab
--principal=oryx/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:37129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=38231
--tserver_master_addrs=127.6.152.62:40931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:16.374307 15650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:16.375239 15650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:16.384539 15658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:16.384735 15650 server_base.cc:1034] running on GCE node
W20260928 23:50:16.384588 15656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:16.384557 15655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:16.385222 15650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:16.385861 15650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:16.387066 15650 hybrid_clock.cc:648] HybridClock initialized: now 1790639416387032 us; error 27 us; skew 500 ppm
Sep 28 23:50:16 dist-test-slave-p5bw krb5kdc[14914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639416, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:16.389691 15650 init.cc:377] Logged in from keytab as oryx/127.6.152.2@KRBTEST.COM (short username oryx)
I20260928 23:50:16.390573 15650 webserver.cc:460] Webserver started at http://127.6.152.2:38231/ using document root <none> and password file <none>
I20260928 23:50:16.391126 15650 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:16.391229 15650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:16.395093 15650 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 23:50:16.397095 15665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:16.398365 15650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:50:16.398488 15650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "6a7ae29fedfa499abc5ec21311d42db3"
format_stamp: "Formatted at 2026-09-28 23:50:15 on dist-test-slave-p5bw"
server_key: "7bacd7c223a2cc53ee3ccdbf5d46fc85"
server_key_iv: "e7395ae8b8af1976992e7dd519465bb8"
server_key_version: "encryptionkey@0"
I20260928 23:50:16.399082 15650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:16.426966 15650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:16.429934 15650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:16.430096 15650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:16.430634 15650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:50:16.431398 15650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:16.431452 15650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:16.431490 15650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:16.431512 15650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:16.440604 15650 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:37129
I20260928 23:50:16.440622 15778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:37129 every 8 connection(s)
I20260928 23:50:16.441699 15650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:50:16.447862 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 15650
I20260928 23:50:16.448355 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:36025
--local_ip_for_outbound_sockets=127.6.152.3
--tserver_master_addrs=127.6.152.62:40931
--webserver_port=43939
--webserver_interface=127.6.152.3
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.152.3
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:50:16 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:16 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 23:50:16.452876 15648 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:16.443431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:34657 (local address 127.6.152.62:40931)
0928 23:50:16.443567 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:16.443570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:16.444371 (+ 801us) server_negotiation.cc:407] Connection header received
0928 23:50:16.445626 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:16.445628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:16.445679 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:16.445753 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:16.446859 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.447368 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.449109 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.449322 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.449403 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:16.452724 (+ 3321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.2:34657: BlockingRecv error: recv got EOF from 127.6.152.2:34657 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:50:16.453506 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:16.443769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.2:34657)
0928 23:50:16.444212 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:16.444253 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:16.445427 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:16.445917 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:16.445924 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:16.446271 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:16.446736 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:16.446749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.447485 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.447488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:16.448990 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:16.449005 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.449472 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.449478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:16.449711 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:16.450363 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:16.450379 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:16.452606 (+ 2227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":107,"threads_started":1}
W20260928 23:50:16.453943 15779 heartbeater.cc:644] Failed to heartbeat to 127.6.152.62:40931 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.152.62:40931: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Sep 28 23:50:16 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:16 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:16.459314 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:16.454409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.2:58921)
0928 23:50:16.454515 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:16.454531 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:16.454610 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:16.454868 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:16.454871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:16.455069 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:16.455156 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:16.455161 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.455867 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.455873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:16.456424 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:16.456431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.456895 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.456898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:16.456996 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:16.457463 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:16.457475 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:16.459202 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 23:50:16.459674 15648 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:16.454487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:58921 (local address 127.6.152.62:40931)
0928 23:50:16.454595 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:16.454599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:16.454612 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:50:16.454721 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:16.454724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:16.454771 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:16.454855 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:16.455268 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.455764 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.456541 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.456760 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.456888 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:16.459338 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.2:58921: BlockingRecv error: recv got EOF from 127.6.152.2:58921 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 23:50:16 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:16 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:16.464874 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:16.459795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.2:52509)
0928 23:50:16.460010 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:16.460022 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:16.460097 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:16.460295 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:16.460297 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:16.460450 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:16.460534 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:16.460540 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.461260 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.461263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:16.462135 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:16.462145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.462534 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.462538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:16.462618 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:16.463023 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:16.463036 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:16.464731 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152}
W20260928 23:50:16.465024 15779 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:50:16.464963 15648 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:16.460004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:52509 (local address 127.6.152.62:40931)
0928 23:50:16.460117 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:16.460120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:16.460132 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:50:16.460163 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:16.460165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:16.460195 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:16.460257 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:16.460636 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.461169 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.462247 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.462442 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.462511 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:16.464863 (+ 2352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.2:52509: BlockingRecv error: recv got EOF from 127.6.152.2:52509 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 23:50:16.559321 15782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:16.559576 15782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:16.559646 15782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:16.563583 15782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:16.563642 15782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:16.563737 15782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:50:16.568887 15782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/krb5kdc/oryx_127.6.152.3.keytab
--principal=oryx/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:36025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=43939
--tserver_master_addrs=127.6.152.62:40931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:16.569909 15782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:16.570897 15782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:16.580102 15788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:16.580138 15787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:16.580245 15782 server_base.cc:1034] running on GCE node
W20260928 23:50:16.580089 15790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:16.580863 15782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:16.581498 15782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:16.582685 15782 hybrid_clock.cc:648] HybridClock initialized: now 1790639416582651 us; error 45 us; skew 500 ppm
Sep 28 23:50:16 dist-test-slave-p5bw krb5kdc[14914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639416, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:16.585323 15782 init.cc:377] Logged in from keytab as oryx/127.6.152.3@KRBTEST.COM (short username oryx)
I20260928 23:50:16.586190 15782 webserver.cc:460] Webserver started at http://127.6.152.3:43939/ using document root <none> and password file <none>
I20260928 23:50:16.586735 15782 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:16.586834 15782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:16.590533 15782 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 23:50:16.592545 15797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:16.593792 15782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 23:50:16.593923 15782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "7503066d06c24264b8ca2d6bb4587f7b"
format_stamp: "Formatted at 2026-09-28 23:50:15 on dist-test-slave-p5bw"
server_key: "43449ef75853c09cdcc162c46d19561f"
server_key_iv: "a838cb14b8d8d66a3df3ccc21bc60859"
server_key_version: "encryptionkey@0"
I20260928 23:50:16.594487 15782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:16.610227 15782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:16.613493 15782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:16.613694 15782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:16.614275 15782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:50:16.615088 15782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:16.615159 15782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:16.615228 15782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:16.615280 15782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:16.624568 15782 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:36025
I20260928 23:50:16.624610 15910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:36025 every 8 connection(s)
I20260928 23:50:16.625628 15782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:50:16.634454 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 15782
Sep 28 23:50:16 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:16 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 23:50:16.636546 15648 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:16.627410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:59033 (local address 127.6.152.62:40931)
0928 23:50:16.627571 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:16.627586 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:16.628596 (+ 1010us) server_negotiation.cc:407] Connection header received
0928 23:50:16.629500 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:16.629503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:16.629550 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:16.629634 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:16.630836 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.631347 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.632441 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.632654 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.632772 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:16.636444 (+ 3672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.3:59033: BlockingRecv error: recv got EOF from 127.6.152.3:59033 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 23:50:16.637373 15914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:16.627771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.3:59033)
0928 23:50:16.628429 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:16.628475 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:16.629311 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:16.629827 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:16.629833 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:16.630204 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:16.630694 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:16.630707 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.631462 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.631465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:16.632336 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:16.632345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.632838 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.632843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:16.633014 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:16.633650 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:16.633666 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:16.636386 (+ 2720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":106,"threads_started":1}
W20260928 23:50:16.637830 15912 heartbeater.cc:644] Failed to heartbeat to 127.6.152.62:40931 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.152.62:40931: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Sep 28 23:50:16 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:16 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:16.644109 15914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:16.638265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.3:49137)
0928 23:50:16.638412 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:16.638424 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:16.638489 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:16.638701 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:16.638705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:16.638886 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:16.638989 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:16.638994 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.639892 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.639897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:16.640872 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:16.640885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.641452 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.641458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:16.641594 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:16.642069 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:16.642086 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:16.643946 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91}
W20260928 23:50:16.644166 15648 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:16.638341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:49137 (local address 127.6.152.62:40931)
0928 23:50:16.638456 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:16.638459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:16.638470 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:50:16.638563 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:16.638565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:16.638598 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:16.638661 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:16.639098 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.639770 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.640971 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.641326 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.641404 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:16.644055 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.3:49137: BlockingRecv error: recv got EOF from 127.6.152.3:49137 (error 108)
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":48}
I20260928 23:50:16.645597 15916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:16.635501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45340 (local address 127.6.152.62:40931)
0928 23:50:16.635737 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:16.635740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:16.635927 (+ 187us) server_negotiation.cc:407] Connection header received
0928 23:50:16.636068 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:16.636070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:16.636110 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:16.636185 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:16.637054 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.637641 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.638866 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.639138 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.639233 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:16.640962 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:16.640969 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:16.640990 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:16.641021 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:16.642993 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:16.643501 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:16.643514 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:16.643520 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:16.643698 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:16.643986 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:16.643991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:16.643992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:16.644589 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:16.644853 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:16.645152 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 23:50:16.645411 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
Sep 28 23:50:16 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:16 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:16.649863 15914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:16.644663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.3:54667)
0928 23:50:16.644856 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:16.644869 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:16.644945 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:16.645180 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:16.645183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:16.645332 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:16.645406 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:16.645411 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.646311 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.646314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:16.646996 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:16.647006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.647453 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.647457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:16.647543 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:16.648013 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:16.648035 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:16.649778 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95}
W20260928 23:50:16.649989 15648 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:16.644765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:54667 (local address 127.6.152.62:40931)
0928 23:50:16.644916 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:16.644919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:16.644931 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:50:16.645014 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:16.645016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:16.645063 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:16.645143 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:16.645508 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.646200 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.647107 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:16.647308 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:16.647439 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:16.649891 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.3:54667: BlockingRecv error: recv got EOF from 127.6.152.3:54667 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260928 23:50:16.650061 15912 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 23:50:17 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:17 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:17.275650 15917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:17.270126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.1:57979)
0928 23:50:17.270383 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:17.270407 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:17.270489 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:17.270738 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:17.270740 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:17.270903 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:17.270995 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:17.271001 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:17.271709 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:17.271711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:17.272393 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:17.272407 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:17.273085 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:17.273091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:17.273176 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:17.273720 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:17.273732 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:17.275524 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":94,"threads_started":1}
W20260928 23:50:17.275784 15918 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:17.270204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:57979 (local address 127.6.152.62:40931)
0928 23:50:17.270442 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:17.270445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:17.270458 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:50:17.270578 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:17.270580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:17.270623 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:17.270700 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:17.271110 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:17.271623 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:17.272528 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:17.272934 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:17.273067 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:17.275653 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.1:57979: BlockingRecv error: recv got EOF from 127.6.152.1:57979 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":100,"threads_started":1}
Sep 28 23:50:17 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:17 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:17.471227 15919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:17.465799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.2:59273)
0928 23:50:17.466056 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:17.466070 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:17.466154 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:17.466421 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:17.466424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:17.466624 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:17.466774 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:17.466783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:17.467691 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:17.467697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:17.468287 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:17.468295 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:17.468704 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:17.468706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:17.468777 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:17.469308 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:17.469319 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:17.471109 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":69,"threads_started":1}
W20260928 23:50:17.471377 15918 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:17.465923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:59273 (local address 127.6.152.62:40931)
0928 23:50:17.466092 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:17.466096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:17.466127 (+ 31us) server_negotiation.cc:407] Connection header received
0928 23:50:17.466258 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:17.466260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:17.466305 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:17.466366 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:17.466910 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:17.467578 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:17.468388 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:17.468603 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:17.468681 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:17.471226 (+ 2545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.2:59273: BlockingRecv error: recv got EOF from 127.6.152.2:59273 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 23:50:17 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:17 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:17.655709 15920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:17.650685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.3:54919)
0928 23:50:17.650941 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:17.650954 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:17.651048 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:17.651274 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:17.651276 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:17.651445 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:17.651553 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:17.651558 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:17.652302 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:17.652304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:17.652900 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:17.652908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:17.653314 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:17.653316 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:17.653385 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:17.653855 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:17.653866 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:17.655592 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":85,"threads_started":1}
W20260928 23:50:17.655956 15918 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:17.650732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:54919 (local address 127.6.152.62:40931)
0928 23:50:17.650877 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:17.650880 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:17.651003 (+ 123us) server_negotiation.cc:407] Connection header received
0928 23:50:17.651124 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:17.651126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:17.651169 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:17.651235 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:17.651667 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:17.652209 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:17.652995 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:17.653206 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:17.653294 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:17.655846 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.3:54919: BlockingRecv error: recv got EOF from 127.6.152.3:54919 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 23:50:18 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:18 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:18.281689 15921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:18.276383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.1:42039)
0928 23:50:18.276661 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:18.276675 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:18.276771 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:18.277030 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:18.277032 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:18.277205 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:18.277309 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:18.277314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:18.278011 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:18.278013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:18.278564 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:18.278570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:18.278954 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:18.278956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:18.279018 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:18.279698 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:18.279709 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:18.281555 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":68,"threads_started":1}
W20260928 23:50:18.281829 15922 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:18.276469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:42039 (local address 127.6.152.62:40931)
0928 23:50:18.276726 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:18.276729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:18.276742 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:50:18.276877 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:18.276879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:18.276924 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:18.276996 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:18.277406 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:18.277927 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:18.278657 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:18.278865 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:18.278935 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:18.281671 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.1:42039: BlockingRecv error: recv got EOF from 127.6.152.1:42039 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":85,"threads_started":1}
Sep 28 23:50:18 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:18 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:18.477489 15923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:18.472155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.2:34603)
0928 23:50:18.472440 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:18.472457 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:18.472582 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:18.472916 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:18.472919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:18.473168 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:18.473317 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:18.473327 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:18.474047 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:18.474050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:18.474597 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:18.474604 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:18.474986 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:18.474988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:18.475046 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:18.475495 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:18.475507 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:18.477357 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":96,"threads_started":1}
W20260928 23:50:18.477635 15922 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:18.472307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:34603 (local address 127.6.152.62:40931)
0928 23:50:18.472441 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:18.472447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:18.472615 (+ 168us) server_negotiation.cc:407] Connection header received
0928 23:50:18.472700 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:18.472702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:18.472750 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:18.472886 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:18.473438 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:18.473954 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:18.474694 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:18.474897 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:18.474969 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:18.477486 (+ 2517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.2:34603: BlockingRecv error: recv got EOF from 127.6.152.2:34603 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 23:50:18 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:18 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:18.662429 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:18.656590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.3:53085)
0928 23:50:18.656940 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:18.656957 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:18.657107 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:18.657386 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:18.657389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:18.657599 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:18.657727 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:18.657734 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:18.658747 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:18.658753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:18.659354 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:18.659362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:18.659775 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:18.659777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:18.659835 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:18.660315 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:18.660329 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:18.662293 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":120,"threads_started":1}
W20260928 23:50:18.662539 15922 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:18.656777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:53085 (local address 127.6.152.62:40931)
0928 23:50:18.656940 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:18.656946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:18.657042 (+ 96us) server_negotiation.cc:407] Connection header received
0928 23:50:18.657206 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:18.657211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:18.657266 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:18.657361 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:18.657854 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:18.658611 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:18.659463 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:18.659680 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:18.659762 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:18.662427 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.3:53085: BlockingRecv error: recv got EOF from 127.6.152.3:53085 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 28 23:50:19 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:19 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:19.287688 15925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:19.282396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.1:37277)
0928 23:50:19.282679 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:19.282692 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:19.282778 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:19.283042 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:19.283045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:19.283219 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:19.283317 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:19.283322 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:19.284038 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:19.284041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:19.284631 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:19.284639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:19.285085 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:19.285088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:19.285154 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:19.285662 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:19.285672 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:19.287539 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":108,"threads_started":1}
W20260928 23:50:19.287822 15926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:19.282495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:37277 (local address 127.6.152.62:40931)
0928 23:50:19.282731 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:19.282734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:19.282747 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:50:19.282841 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:19.282843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:19.282896 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:19.283021 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:19.283410 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:19.283920 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:19.284730 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:19.284990 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:19.285068 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:19.287682 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.1:37277: BlockingRecv error: recv got EOF from 127.6.152.1:37277 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":104,"threads_started":1}
Sep 28 23:50:19 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:19 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:19.483590 15927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:19.478155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.2:34507)
0928 23:50:19.478421 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:19.478434 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:19.478519 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:19.478800 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:19.478802 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:19.479003 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:19.479097 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:19.479103 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:19.479883 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:19.479886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:19.480664 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:19.480673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:19.481199 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:19.481202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:19.481263 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:19.481715 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:19.481726 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:19.483478 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":107,"threads_started":1}
W20260928 23:50:19.483733 15926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:19.478316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:34507 (local address 127.6.152.62:40931)
0928 23:50:19.478448 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:19.478452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:19.478479 (+ 27us) server_negotiation.cc:407] Connection header received
0928 23:50:19.478604 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:19.478606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:19.478650 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:19.478724 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:19.479215 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:19.479752 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:19.480783 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:19.481095 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:19.481176 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:19.483619 (+ 2443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.2:34507: BlockingRecv error: recv got EOF from 127.6.152.2:34507 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 23:50:19 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:19 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:19.669076 15928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:19.663334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.3:42097)
0928 23:50:19.663635 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:19.663648 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:19.663781 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:19.664102 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:19.664105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:19.664363 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:19.664473 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:19.664478 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:19.665477 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:19.665480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:19.666042 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:19.666051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:19.666504 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:19.666507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:19.666578 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:19.667052 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:19.667064 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:19.668875 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":124,"threads_started":1}
W20260928 23:50:19.669147 15926 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:19.663408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:42097 (local address 127.6.152.62:40931)
0928 23:50:19.663613 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:19.663617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:19.663737 (+ 120us) server_negotiation.cc:407] Connection header received
0928 23:50:19.663905 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:19.663908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:19.663965 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:19.664063 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:19.664646 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:19.665379 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:19.666172 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:19.666401 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:19.666493 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:19.669051 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.3:42097: BlockingRecv error: recv got EOF from 127.6.152.3:42097 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 28 23:50:20 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:20 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:20.297874 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:20.288546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.1:54307)
0928 23:50:20.288905 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:20.288928 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:20.289058 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:20.289383 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:20.289386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:20.289623 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:20.289748 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:20.289754 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:20.290571 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:20.290573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:20.291155 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:20.291163 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:20.291604 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:20.291607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:20.291676 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:20.295873 (+ 4197us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:20.295887 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:20.297745 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":81,"threads_started":1}
W20260928 23:50:20.298154 15930 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:20.288685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:54307 (local address 127.6.152.62:40931)
0928 23:50:20.288988 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:20.288994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:20.289013 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:50:20.289169 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:20.289174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:20.289238 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:20.289325 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:20.289917 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:20.290473 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:20.291258 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:20.291483 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:20.291564 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:20.297926 (+ 6362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.1:54307: BlockingRecv error: recv got EOF from 127.6.152.1:54307 (error 108)
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":56,"threads_started":1}
Sep 28 23:50:20 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:20 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:20.489897 15931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:20.484508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.2:56577)
0928 23:50:20.484884 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:20.484904 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:20.485035 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:20.485326 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:20.485329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:20.485541 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:20.485641 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:20.485647 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:20.486405 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:20.486410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:20.486970 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:20.486978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:20.487437 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:20.487441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:20.487527 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:20.488007 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:20.488017 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:20.489783 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":104,"threads_started":1}
W20260928 23:50:20.490111 15930 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:20.484611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:56577 (local address 127.6.152.62:40931)
0928 23:50:20.484750 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:20.484755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:20.484983 (+ 228us) server_negotiation.cc:407] Connection header received
0928 23:50:20.485154 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:20.485157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:20.485207 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:20.485304 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:20.485761 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:20.486293 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:20.487094 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:20.487327 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:20.487436 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:20.489958 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.2:56577: BlockingRecv error: recv got EOF from 127.6.152.2:56577 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 23:50:20 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:20 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:20.675439 15932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:20.669939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.3:39995)
0928 23:50:20.670204 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:20.670217 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:20.670351 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:20.670627 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:20.670630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:20.670834 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:20.670934 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:20.670940 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:20.671727 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:20.671732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:20.672316 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:20.672324 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:20.672852 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:20.672856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:20.672962 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:20.673425 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:20.673436 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:20.675323 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":95,"threads_started":1}
W20260928 23:50:20.675575 15930 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:20.669999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:39995 (local address 127.6.152.62:40931)
0928 23:50:20.670145 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:20.670149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:20.670297 (+ 148us) server_negotiation.cc:407] Connection header received
0928 23:50:20.670465 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:20.670470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:20.670527 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:20.670604 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:20.671076 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:20.671615 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:20.672477 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:20.672705 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:20.672816 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:20.675462 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.3:39995: BlockingRecv error: recv got EOF from 127.6.152.3:39995 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 23:50:21 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:21 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:21.304448 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:21.298697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.1:58755)
0928 23:50:21.298994 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:21.299008 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:21.299138 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:21.299398 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:21.299401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:21.299622 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:21.299713 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:21.299719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:21.300545 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:21.300550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:21.301197 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:21.301206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:21.301669 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:21.301673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:21.301756 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:21.302318 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:21.302332 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:21.304337 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":97,"threads_started":1}
W20260928 23:50:21.304625 15934 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:21.298736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:58755 (local address 127.6.152.62:40931)
0928 23:50:21.299020 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:21.299023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:21.299103 (+ 80us) server_negotiation.cc:407] Connection header received
0928 23:50:21.299222 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:21.299224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:21.299264 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:21.299377 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:21.299833 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:21.300441 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:21.301315 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:21.301549 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:21.301658 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:21.304479 (+ 2821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.1:58755: BlockingRecv error: recv got EOF from 127.6.152.1:58755 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":101,"threads_started":1}
Sep 28 23:50:21 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:21 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:21.495954 15935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:21.490693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.2:37979)
0928 23:50:21.490947 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:21.490960 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:21.491041 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:21.491252 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:21.491254 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:21.491432 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:21.491521 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:21.491525 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:21.492361 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:21.492363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:21.493037 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:21.493046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:21.493491 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:21.493493 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:21.493554 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:21.494072 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:21.494083 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:21.495825 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":82,"threads_started":1}
W20260928 23:50:21.496100 15934 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:21.490834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:37979 (local address 127.6.152.62:40931)
0928 23:50:21.490989 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:21.490993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:21.491007 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:50:21.491125 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:21.491127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:21.491171 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:21.491235 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:21.491645 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:21.492255 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:21.493160 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:21.493386 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:21.493478 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:21.495966 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.2:37979: BlockingRecv error: recv got EOF from 127.6.152.2:37979 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 23:50:21 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:21 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:21.681901 15936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:21.676396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.3:34895)
0928 23:50:21.676671 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:21.676686 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:21.676816 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:21.677094 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:21.677096 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:21.677302 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:21.677392 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:21.677397 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:21.678156 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:21.678161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:21.678740 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:21.678748 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:21.679202 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:21.679206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:21.679289 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:21.679822 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:21.679842 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:21.681776 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":88,"threads_started":1}
W20260928 23:50:21.682035 15934 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:21.676458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:34895 (local address 127.6.152.62:40931)
0928 23:50:21.676617 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:21.676622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:21.676764 (+ 142us) server_negotiation.cc:407] Connection header received
0928 23:50:21.676919 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:21.676923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:21.676979 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:21.677062 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:21.677507 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:21.678044 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:21.678861 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:21.679092 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:21.679198 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:21.681917 (+ 2719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.3:34895: BlockingRecv error: recv got EOF from 127.6.152.3:34895 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 23:50:22 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:22 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:22.310843 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:22.305272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.1:34737)
0928 23:50:22.305670 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:22.305686 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:22.305811 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:22.306162 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:22.306165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:22.306418 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:22.306546 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:22.306554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:22.307344 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:22.307346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:22.307901 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:22.307908 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:22.308337 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:22.308340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:22.308400 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:22.308936 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:22.308947 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:22.310736 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":113,"threads_started":1}
W20260928 23:50:22.311039 15938 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:22.305320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:34737 (local address 127.6.152.62:40931)
0928 23:50:22.305588 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:22.305592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:22.305788 (+ 196us) server_negotiation.cc:407] Connection header received
0928 23:50:22.305947 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:22.305950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:22.306007 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:22.306111 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:22.306692 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:22.307250 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:22.308007 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:22.308245 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:22.308322 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:22.310868 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.1:34737: BlockingRecv error: recv got EOF from 127.6.152.1:34737 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":101,"threads_started":1}
Sep 28 23:50:22 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:22 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:22.502605 15939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:22.496927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.2:39523)
0928 23:50:22.497232 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:22.497244 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:22.497369 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:22.497658 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:22.497660 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:22.497840 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:22.497933 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:22.497939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:22.498782 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:22.498785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:22.499357 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:22.499413 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:22.499876 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:22.499879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:22.499968 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:22.500466 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:22.500478 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:22.502436 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":124,"threads_started":1}
W20260928 23:50:22.502713 15938 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:22.496972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:39523 (local address 127.6.152.62:40931)
0928 23:50:22.497159 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:22.497162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:22.497392 (+ 230us) server_negotiation.cc:407] Connection header received
0928 23:50:22.497503 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:22.497505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:22.497549 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:22.497623 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:22.498031 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:22.498673 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:22.499536 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:22.499760 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:22.499870 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:22.502569 (+ 2699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.2:39523: BlockingRecv error: recv got EOF from 127.6.152.2:39523 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 28 23:50:22 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:22 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:22.688465 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:22.682821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.3:43727)
0928 23:50:22.683161 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:22.683178 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:22.683301 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:22.683586 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:22.683589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:22.683798 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:22.683886 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:22.683891 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:22.684649 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:22.684655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:22.685238 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:22.685246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:22.685715 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:22.685719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:22.685805 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:22.686284 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:22.686295 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:22.688349 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":79,"threads_started":1}
W20260928 23:50:22.688637 15938 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:22.683071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:43727 (local address 127.6.152.62:40931)
0928 23:50:22.683219 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:22.683226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:22.683242 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:50:22.683416 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:22.683419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:22.683473 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:22.683556 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:22.684006 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:22.684533 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:22.685357 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:22.685603 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:22.685713 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:22.688472 (+ 2759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.3:43727: BlockingRecv error: recv got EOF from 127.6.152.3:43727 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 23:50:23 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:23 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:23.317185 15941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:23.311586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.1:42113)
0928 23:50:23.311884 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:23.311898 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:23.312018 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:23.312322 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:23.312328 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:23.312565 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:23.312711 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:23.312719 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:23.313735 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:23.313738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:23.314303 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:23.314311 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:23.314710 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:23.314712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:23.314774 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:23.315247 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:23.315258 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:23.317072 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":89,"threads_started":1}
W20260928 23:50:23.317371 15942 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:23.311697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:42113 (local address 127.6.152.62:40931)
0928 23:50:23.311914 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:23.311918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:23.311993 (+ 75us) server_negotiation.cc:407] Connection header received
0928 23:50:23.312111 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:23.312114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:23.312167 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:23.312274 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:23.312860 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:23.313636 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:23.314396 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:23.314621 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:23.314693 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:23.317203 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.1:42113: BlockingRecv error: recv got EOF from 127.6.152.1:42113 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":79,"threads_started":1}
Sep 28 23:50:23 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:23 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:23.508754 15943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:23.503480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.2:52161)
0928 23:50:23.503822 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:23.503835 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:23.503930 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:23.504164 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:23.504166 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:23.504342 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:23.504432 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:23.504438 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:23.505152 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:23.505155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:23.505696 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:23.505704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:23.506076 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:23.506078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:23.506138 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:23.506602 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:23.506613 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:23.508643 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":113,"threads_started":1}
W20260928 23:50:23.508939 15942 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:23.503524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:52161 (local address 127.6.152.62:40931)
0928 23:50:23.503681 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:23.503686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:23.503902 (+ 216us) server_negotiation.cc:407] Connection header received
0928 23:50:23.504005 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:23.504007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:23.504070 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:23.504137 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:23.504521 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:23.505067 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:23.505786 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:23.505993 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:23.506079 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:23.508760 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.2:52161: BlockingRecv error: recv got EOF from 127.6.152.2:52161 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 23:50:23 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:23 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:23.694927 15944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:23.689291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.3:47553)
0928 23:50:23.689594 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:23.689606 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:23.689702 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:23.689936 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:23.689938 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:23.690125 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:23.690218 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:23.690223 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:23.691032 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:23.691034 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:23.691835 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:23.691844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:23.692319 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:23.692322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:23.692407 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:23.692990 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:23.693004 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:23.694805 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":78,"threads_started":1}
W20260928 23:50:23.695101 15942 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:23.689351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:47553 (local address 127.6.152.62:40931)
0928 23:50:23.689470 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:23.689474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:23.689668 (+ 194us) server_negotiation.cc:407] Connection header received
0928 23:50:23.689767 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:23.689769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:23.689814 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:23.689898 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:23.690307 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:23.690928 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:23.691976 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:23.692193 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:23.692312 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:23.694946 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.3:47553: BlockingRecv error: recv got EOF from 127.6.152.3:47553 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 23:50:24 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:24 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:24.323240 15945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:24.317849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.1:36825)
0928 23:50:24.318223 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:24.318236 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:24.318336 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:24.318565 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:24.318567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:24.318743 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:24.318840 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:24.318845 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:24.319570 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:24.319573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:24.320136 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:24.320144 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:24.320541 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:24.320543 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:24.320605 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:24.321175 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:24.321187 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:24.323069 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":113,"threads_started":1}
W20260928 23:50:24.323329 15946 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:24.317993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:36825 (local address 127.6.152.62:40931)
0928 23:50:24.318219 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:24.318222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:24.318301 (+ 79us) server_negotiation.cc:407] Connection header received
0928 23:50:24.318412 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:24.318414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:24.318456 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:24.318530 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:24.318930 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:24.319474 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:24.320234 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:24.320452 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:24.320525 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:24.323201 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.1:36825: BlockingRecv error: recv got EOF from 127.6.152.1:36825 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":71,"threads_started":1}
Sep 28 23:50:24 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:24 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:24.514853 15947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:24.509514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.2:54095)
0928 23:50:24.509769 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:24.509784 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:24.509886 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:24.510123 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:24.510126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:24.510302 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:24.510393 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:24.510398 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:24.511325 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:24.511327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:24.511872 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:24.511880 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:24.512358 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:24.512361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:24.512422 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:24.512966 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:24.512977 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:24.514741 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":96,"threads_started":1}
W20260928 23:50:24.515020 15946 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:24.509597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:54095 (local address 127.6.152.62:40931)
0928 23:50:24.509731 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:24.509734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:24.509892 (+ 158us) server_negotiation.cc:407] Connection header received
0928 23:50:24.509949 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:24.509952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:24.510005 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:24.510090 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:24.510493 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:24.511221 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:24.511990 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:24.512243 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:24.512347 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:24.514867 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.2:54095: BlockingRecv error: recv got EOF from 127.6.152.2:54095 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 23:50:24 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:24 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:24.701260 15948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:24.695638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.3:51283)
0928 23:50:24.695929 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:24.695942 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:24.696072 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:24.696320 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:24.696323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:24.696530 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:24.696619 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:24.696624 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:24.697418 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:24.697424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:24.698086 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:24.698095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:24.698562 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:24.698566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:24.698643 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:24.699225 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:24.699236 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:24.701149 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":83,"threads_started":1}
W20260928 23:50:24.701431 15946 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:24.695721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:51283 (local address 127.6.152.62:40931)
0928 23:50:24.695849 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:24.695853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:24.696008 (+ 155us) server_negotiation.cc:407] Connection header received
0928 23:50:24.696178 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:24.696181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:24.696225 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:24.696302 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:24.696750 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:24.697310 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:24.698200 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:24.698441 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:24.698568 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:24.701278 (+ 2710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.3:51283: BlockingRecv error: recv got EOF from 127.6.152.3:51283 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 23:50:25 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:25 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:25.335302 15949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:25.323936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.1:37981)
0928 23:50:25.324213 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:25.324227 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:25.324316 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:25.324559 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:25.324561 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:25.324736 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:25.324902 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:25.324909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:25.325615 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:25.325617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:25.326170 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:25.326178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:25.326558 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:25.326560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:25.326618 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:25.333052 (+ 6434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:25.333064 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:25.335133 (+ 2069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":92,"threads_started":1}
W20260928 23:50:25.335383 15950 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:25.324010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:37981 (local address 127.6.152.62:40931)
0928 23:50:25.324254 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:25.324258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:25.324271 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:50:25.324403 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:25.324405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:25.324451 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:25.324528 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:25.324999 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:25.325531 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:25.326260 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:25.326472 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:25.326544 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:25.335258 (+ 8714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.1:37981: BlockingRecv error: recv got EOF from 127.6.152.1:37981 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":59,"threads_started":1}
Sep 28 23:50:25 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:25 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:25.521119 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:25.515562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.2:37209)
0928 23:50:25.515841 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:25.515854 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:25.515956 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:25.516169 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:25.516170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:25.516350 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:25.516447 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:25.516452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:25.517336 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:25.517338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:25.517967 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:25.518001 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:25.518447 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:25.518450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:25.518517 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:25.519068 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:25.519080 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:25.521005 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":112,"threads_started":1}
W20260928 23:50:25.521281 15950 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:25.515669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:37209 (local address 127.6.152.62:40931)
0928 23:50:25.515809 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:25.515812 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:25.515953 (+ 141us) server_negotiation.cc:407] Connection header received
0928 23:50:25.516022 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:25.516024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:25.516067 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:25.516135 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:25.516573 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:25.517215 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:25.518106 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:25.518324 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:25.518406 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:25.521130 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.2:37209: BlockingRecv error: recv got EOF from 127.6.152.2:37209 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 23:50:25 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:25 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:25.707561 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:25.701982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.3:40787)
0928 23:50:25.702334 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:25.702346 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:25.702442 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:25.702686 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:25.702692 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:25.702875 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:25.702979 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:25.702985 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:25.703810 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:25.703815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:25.704372 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:25.704380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:25.704890 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:25.704893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:25.704976 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:25.705428 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:25.705439 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:25.707414 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":83,"threads_started":1}
W20260928 23:50:25.707739 15950 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:25.702079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:40787 (local address 127.6.152.62:40931)
0928 23:50:25.702194 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:25.702198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:25.702417 (+ 219us) server_negotiation.cc:407] Connection header received
0928 23:50:25.702523 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:25.702525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:25.702572 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:25.702633 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:25.703124 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:25.703675 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:25.704488 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:25.704728 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:25.704888 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:25.707626 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.3:40787: BlockingRecv error: recv got EOF from 127.6.152.3:40787 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 23:50:26 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:26 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:26.341529 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:26.336217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.1:46661)
0928 23:50:26.336476 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:26.336490 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:26.336579 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:26.336867 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:26.336869 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:26.337043 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:26.337159 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:26.337164 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:26.337889 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:26.337891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:26.338455 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:26.338463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:26.338860 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:26.338862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:26.338925 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:26.339406 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:26.339417 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:26.341358 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":85,"threads_started":1}
W20260928 23:50:26.341591 15954 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:26.336246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:46661 (local address 127.6.152.62:40931)
0928 23:50:26.336504 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:26.336507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:26.336552 (+ 45us) server_negotiation.cc:407] Connection header received
0928 23:50:26.336666 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:26.336669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:26.336713 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:26.336793 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:26.337262 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:26.337795 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:26.338560 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:26.338771 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:26.338844 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:26.341475 (+ 2631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.1:46661: BlockingRecv error: recv got EOF from 127.6.152.1:46661 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":100,"threads_started":1}
Sep 28 23:50:26 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:26 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:26.527609 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:26.521908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.2:48767)
0928 23:50:26.522203 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:26.522216 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:26.522340 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:26.522593 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:26.522597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:26.522802 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:26.522902 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:26.522908 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:26.523666 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:26.523672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:26.524230 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:26.524238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:26.524829 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:26.524833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:26.524921 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:26.525512 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:26.525527 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:26.527447 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":85,"threads_started":1}
W20260928 23:50:26.527670 15954 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:26.521975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:48767 (local address 127.6.152.62:40931)
0928 23:50:26.522138 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:26.522144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:26.522287 (+ 143us) server_negotiation.cc:407] Connection header received
0928 23:50:26.522435 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:26.522440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:26.522490 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:26.522571 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:26.523025 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:26.523555 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:26.524342 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:26.524707 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:26.524826 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:26.527558 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.2:48767: BlockingRecv error: recv got EOF from 127.6.152.2:48767 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 23:50:26 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:26 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:26.714176 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:26.708574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.3:52491)
0928 23:50:26.709031 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:26.709046 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:26.709184 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:26.709502 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:26.709506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:26.709769 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:26.709898 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:26.709905 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:26.710677 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:26.710680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:26.711283 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:26.711292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:26.711696 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:26.711698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:26.711787 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:26.712255 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:26.712268 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:26.714059 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":117,"threads_started":1}
W20260928 23:50:26.714318 15954 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:26.708723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:52491 (local address 127.6.152.62:40931)
0928 23:50:26.708879 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:26.708882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:26.709146 (+ 264us) server_negotiation.cc:407] Connection header received
0928 23:50:26.709286 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:26.709291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:26.709350 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:26.709438 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:26.710057 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:26.710573 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:26.711380 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:26.711590 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:26.711664 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:26.714203 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.3:52491: BlockingRecv error: recv got EOF from 127.6.152.3:52491 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 23:50:27 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:27 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:27.347683 15957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:27.342471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.1:37839)
0928 23:50:27.342774 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:27.342792 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:27.342911 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:27.343176 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:27.343178 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:27.343350 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:27.343446 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:27.343450 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:27.344180 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:27.344182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:27.344721 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:27.344729 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:27.345179 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:27.345182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:27.345245 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:27.345738 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:27.345749 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:27.347542 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":72,"threads_started":1}
W20260928 23:50:27.347823 15958 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:27.342538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:37839 (local address 127.6.152.62:40931)
0928 23:50:27.342803 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:27.342807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:27.342867 (+ 60us) server_negotiation.cc:407] Connection header received
0928 23:50:27.343022 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:27.343025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:27.343069 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:27.343145 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:27.343537 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:27.344094 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:27.344838 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:27.345087 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:27.345163 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:27.347667 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.1:37839: BlockingRecv error: recv got EOF from 127.6.152.1:37839 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":91,"threads_started":1}
Sep 28 23:50:27 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:27 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:27.534382 15959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:27.528372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.2:55633)
0928 23:50:27.528698 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:27.528717 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:27.528881 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:27.529188 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:27.529191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:27.529408 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:27.529513 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:27.529519 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:27.530278 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:27.530284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:27.530843 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:27.530851 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:27.531313 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:27.531316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:27.531415 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:27.531931 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:27.531943 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:27.534214 (+ 2271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":82,"threads_started":1}
W20260928 23:50:27.534480 15958 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:27.528496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:55633 (local address 127.6.152.62:40931)
0928 23:50:27.528657 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:27.528662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:27.528791 (+ 129us) server_negotiation.cc:407] Connection header received
0928 23:50:27.529023 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:27.529026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:27.529073 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:27.529154 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:27.529631 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:27.530173 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:27.530976 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:27.531203 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:27.531309 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:27.534349 (+ 3040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.2:55633: BlockingRecv error: recv got EOF from 127.6.152.2:55633 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 23:50:27 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:27 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:27.720496 15960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:27.715075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.3:32889)
0928 23:50:27.715339 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:27.715351 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:27.715473 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:27.715748 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:27.715751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:27.715956 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:27.716043 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:27.716048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:27.716881 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:27.716885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:27.717551 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:27.717560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:27.717988 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:27.717992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:27.718077 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:27.718578 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:27.718590 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:27.720335 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
W20260928 23:50:27.720551 15958 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:27.715113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:32889 (local address 127.6.152.62:40931)
0928 23:50:27.715264 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:27.715269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:27.715422 (+ 153us) server_negotiation.cc:407] Connection header received
0928 23:50:27.715600 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:27.715603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:27.715648 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:27.715729 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:27.716154 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:27.716738 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:27.717661 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:27.717884 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:27.717988 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:27.720447 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.3:32889: BlockingRecv error: recv got EOF from 127.6.152.3:32889 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 23:50:28 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:28 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:28.353654 15961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:28.348494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.1:44823)
0928 23:50:28.348762 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:28.348778 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:28.348903 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:28.349170 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:28.349172 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:28.349346 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:28.349444 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:28.349449 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:28.350181 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:28.350184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:28.350737 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:28.350745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:28.351140 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:28.351142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:28.351206 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:28.351687 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:28.351698 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:28.353488 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":98,"threads_started":1}
W20260928 23:50:28.353753 15962 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:28.348567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:44823 (local address 127.6.152.62:40931)
0928 23:50:28.348854 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:28.348858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:28.348877 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:50:28.349011 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:28.349014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:28.349058 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:28.349139 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:28.349537 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:28.350066 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:28.350837 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:28.351051 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:28.351123 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:28.353620 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.1:44823: BlockingRecv error: recv got EOF from 127.6.152.1:44823 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":85,"threads_started":1}
Sep 28 23:50:28 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:28 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:28.540987 15963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:28.535272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.2:54417)
0928 23:50:28.535580 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:28.535594 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:28.535720 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:28.535986 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:28.535989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:28.536217 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:28.536311 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:28.536317 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:28.537178 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:28.537182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:28.537772 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:28.537780 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:28.538240 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:28.538243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:28.538328 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:28.538812 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:28.538823 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:28.540833 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":108,"threads_started":1}
W20260928 23:50:28.541071 15962 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:28.535310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:54417 (local address 127.6.152.62:40931)
0928 23:50:28.535468 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:28.535480 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:28.535667 (+ 187us) server_negotiation.cc:407] Connection header received
0928 23:50:28.535818 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:28.535823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:28.535878 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:28.535959 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:28.536431 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:28.537061 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:28.537885 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:28.538131 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:28.538236 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:28.540984 (+ 2748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.2:54417: BlockingRecv error: recv got EOF from 127.6.152.2:54417 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 23:50:28 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:28 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:28.726900 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:28.721224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.3:51599)
0928 23:50:28.721493 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:28.721509 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:28.721612 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:28.721868 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:28.721870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:28.722056 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:28.722151 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:28.722156 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:28.723028 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:28.723032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:28.723756 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:28.723765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:28.724255 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:28.724258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:28.724318 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:28.724794 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:28.724860 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:28.726725 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":73,"threads_started":1}
W20260928 23:50:28.726954 15962 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:28.721351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:51599 (local address 127.6.152.62:40931)
0928 23:50:28.721518 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:28.721521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:28.721580 (+ 59us) server_negotiation.cc:407] Connection header received
0928 23:50:28.721690 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:28.721693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:28.721739 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:28.721805 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:28.722281 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:28.722917 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:28.723878 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:28.724152 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:28.724229 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:28.726845 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.3:51599: BlockingRecv error: recv got EOF from 127.6.152.3:51599 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 28 23:50:29 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:29 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:29.359624 15965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:29.354451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.1:55061)
0928 23:50:29.354726 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:29.354739 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:29.354832 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:29.355054 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:29.355056 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:29.355245 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:29.355339 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:29.355344 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:29.356052 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:29.356054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:29.356629 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:29.356636 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:29.357138 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:29.357140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:29.357201 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:29.357729 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:29.357741 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:29.359500 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":81,"threads_started":1}
W20260928 23:50:29.359800 15966 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:29.354508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:55061 (local address 127.6.152.62:40931)
0928 23:50:29.354773 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:29.354776 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:29.354789 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:50:29.354898 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:29.354900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:29.354942 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:29.355041 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:29.355432 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:29.355948 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:29.356719 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:29.357028 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:29.357105 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:29.359634 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.1:55061: BlockingRecv error: recv got EOF from 127.6.152.1:55061 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":96,"threads_started":1}
Sep 28 23:50:29 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:29 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:29.547209 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:29.541799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.2:41829)
0928 23:50:29.542089 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:29.542119 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:29.542213 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:29.542419 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:29.542421 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:29.542611 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:29.542708 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:29.542713 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:29.543523 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:29.543526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:29.544121 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:29.544130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:29.544539 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:29.544541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:29.544601 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:29.545182 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:29.545193 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:29.547084 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":114,"threads_started":1}
W20260928 23:50:29.547315 15966 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:29.541907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:41829 (local address 127.6.152.62:40931)
0928 23:50:29.542058 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:29.542062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:29.542169 (+ 107us) server_negotiation.cc:407] Connection header received
0928 23:50:29.542278 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:29.542280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:29.542323 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:29.542388 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:29.542836 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:29.543385 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:29.544228 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:29.544445 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:29.544521 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:29.547207 (+ 2686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.2:41829: BlockingRecv error: recv got EOF from 127.6.152.2:41829 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 23:50:29 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:29 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:29.733035 15968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:29.727753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.3:36541)
0928 23:50:29.728064 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:29.728076 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:29.728180 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:29.728401 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:29.728403 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:29.728580 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:29.728673 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:29.728678 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:29.729480 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:29.729482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:29.730028 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:29.730035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:29.730455 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:29.730457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:29.730517 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:29.731039 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:29.731051 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:29.732865 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":86,"threads_started":1}
W20260928 23:50:29.733076 15966 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:29.727792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:36541 (local address 127.6.152.62:40931)
0928 23:50:29.727929 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:29.727933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:29.728143 (+ 210us) server_negotiation.cc:407] Connection header received
0928 23:50:29.728244 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:29.728246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:29.728298 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:29.728363 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:29.728791 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:29.729378 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:29.730136 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:29.730353 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:29.730434 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:29.732977 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.3:36541: BlockingRecv error: recv got EOF from 127.6.152.3:36541 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 23:50:30 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:30 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:30.370929 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:30.360393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.1:60673)
0928 23:50:30.360668 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:30.360682 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:30.360767 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:30.361107 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:30.361109 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:30.361285 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:30.361381 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:30.361386 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:30.362080 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:30.362083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:30.362651 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:30.362659 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:30.363061 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:30.363063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:30.363125 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:30.368953 (+ 5828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:30.368965 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:30.370804 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":110,"threads_started":1}
W20260928 23:50:30.371192 15970 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:30.360623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:60673 (local address 127.6.152.62:40931)
0928 23:50:30.360908 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:30.360911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:30.360923 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:50:30.360955 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:30.360957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:30.360994 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:30.361071 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:30.361475 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:30.361993 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:30.362744 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:30.362973 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:30.363045 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:30.370955 (+ 7910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.1:60673: BlockingRecv error: recv got EOF from 127.6.152.1:60673 (error 108)
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":74,"threads_started":1}
Sep 28 23:50:30 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:30 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:30.553670 15971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:30.547982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.2:59255)
0928 23:50:30.548320 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:30.548333 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:30.548466 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:30.548725 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:30.548729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:30.549031 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:30.549139 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:30.549146 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:30.549919 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:30.549923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:30.550520 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:30.550528 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:30.550980 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:30.550984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:30.551067 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:30.551611 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:30.551625 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:30.553495 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":108,"threads_started":1}
W20260928 23:50:30.553721 15970 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:30.548018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:59255 (local address 127.6.152.62:40931)
0928 23:50:30.548167 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:30.548172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:30.548413 (+ 241us) server_negotiation.cc:407] Connection header received
0928 23:50:30.548571 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:30.548574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:30.548625 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:30.548705 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:30.549256 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:30.549811 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:30.550642 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:30.550865 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:30.550969 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:30.553618 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.2:59255: BlockingRecv error: recv got EOF from 127.6.152.2:59255 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 23:50:30 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:30 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.3@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:30.739281 15972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:30.733909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.3:40653)
0928 23:50:30.734225 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:30.734237 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:30.734369 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:30.734629 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:30.734632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:30.734839 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:30.734928 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:30.734933 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:30.735665 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:30.735670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:30.736227 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:30.736235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:30.736683 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:30.736687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:30.736773 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:30.737356 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:30.737368 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:30.739111 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":119,"threads_started":1}
W20260928 23:50:30.739337 15970 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:30.733967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:40653 (local address 127.6.152.62:40931)
0928 23:50:30.734121 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:30.734126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:30.734317 (+ 191us) server_negotiation.cc:407] Connection header received
0928 23:50:30.734466 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:30.734470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:30.734523 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:30.734616 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:30.735043 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:30.735560 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:30.736344 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:30.736569 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:30.736682 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:30.739231 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.3:40653: BlockingRecv error: recv got EOF from 127.6.152.3:40653 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 23:50:31 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:31 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.1@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:31.377106 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:31.372031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.1:42263)
0928 23:50:31.372309 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:31.372322 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:31.372402 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:31.372628 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:31.372630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:31.372858 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:31.372957 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:31.372963 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:31.373691 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:31.373694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:31.374256 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:31.374264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:31.374664 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:31.374667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:31.374726 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:31.375239 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:31.375251 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:31.376987 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":94,"threads_started":1}
W20260928 23:50:31.377255 15974 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:31.372103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:42263 (local address 127.6.152.62:40931)
0928 23:50:31.372375 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:31.372378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:31.372391 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:50:31.372473 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:31.372475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:31.372518 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:31.372595 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:31.373071 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:31.373601 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:31.374354 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:31.374576 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:31.374652 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:31.377112 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.1:42263: BlockingRecv error: recv got EOF from 127.6.152.1:42263 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":132,"threads_started":1}
Sep 28 23:50:31 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:50:31 dist-test-slave-p5bw krb5kdc[14914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.152.2@KRBTEST.COM for oryx/127.6.152.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:50:31.559756 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:31.554717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40931 (local address 127.6.152.2:51281)
0928 23:50:31.554979 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:31.554993 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:31.555129 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:31.555442 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:31.555444 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:31.555626 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:31.555720 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:31.555724 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:31.556417 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:31.556419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:31.557032 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:31.557040 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:31.557413 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:31.557416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:31.557471 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:31.557927 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:31.557938 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:31.559645 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.152.62:40931: Server oryx/127.6.152.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":85,"threads_started":1}
W20260928 23:50:31.559877 15974 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:31.554754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:51281 (local address 127.6.152.62:40931)
0928 23:50:31.554906 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:31.554911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:31.555084 (+ 173us) server_negotiation.cc:407] Connection header received
0928 23:50:31.555217 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:31.555243 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:31.555296 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:31.555390 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:31.555815 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:31.556331 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:31.557120 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:31.557327 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:31.557402 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:31.559765 (+ 2363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.152.2:51281: BlockingRecv error: recv got EOF from 127.6.152.2:51281 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 23:50:31.642311 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 15498
I20260928 23:50:31.647560 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 15650
I20260928 23:50:31.652541 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 15782
I20260928 23:50:31.657428 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 15428
2026-09-28T23:50:31Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17959 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T23:50:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:50:31Z Disabled control of system clock
I20260928 23:50:31.686653 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:46833
--webserver_interface=127.6.152.62
--webserver_port=0
--builtin_ntp_servers=127.6.152.20:37803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:46833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 23:50:31.793658 15979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:31.793905 15979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:31.793937 15979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:31.797802 15979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:50:31.797847 15979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:31.797863 15979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:31.797878 15979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:50:31.797894 15979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:50:31.802520 15979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:37803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:46833
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:46833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:31.803473 15979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:31.804765 15979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:50:31.813179 15979 server_base.cc:1034] running on GCE node
W20260928 23:50:31.813271 15987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:31.813284 15985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:31.813156 15984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:31.814029 15979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:31.815040 15979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:31.816215 15979 hybrid_clock.cc:648] HybridClock initialized: now 1790639431816200 us; error 26 us; skew 500 ppm
I20260928 23:50:31.817947 15979 webserver.cc:460] Webserver started at http://127.6.152.62:40475/ using document root <none> and password file <none>
I20260928 23:50:31.818490 15979 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:31.818576 15979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:31.818795 15979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:31.820760 15979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "7ca6bd2c1c6d4e0cad0e63d365f667c1"
format_stamp: "Formatted at 2026-09-28 23:50:31 on dist-test-slave-p5bw"
server_key: "ec06281eb4fbf32d5a457d3c61c80b6f"
server_key_iv: "0dbb5314152306884c1d55ecb8080c31"
server_key_version: "encryptionkey@0"
I20260928 23:50:31.821393 15979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "7ca6bd2c1c6d4e0cad0e63d365f667c1"
format_stamp: "Formatted at 2026-09-28 23:50:31 on dist-test-slave-p5bw"
server_key: "ec06281eb4fbf32d5a457d3c61c80b6f"
server_key_iv: "0dbb5314152306884c1d55ecb8080c31"
server_key_version: "encryptionkey@0"
I20260928 23:50:31.824780 15979 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:50:31.827035 15993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:31.828120 15979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:50:31.828244 15979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "7ca6bd2c1c6d4e0cad0e63d365f667c1"
format_stamp: "Formatted at 2026-09-28 23:50:31 on dist-test-slave-p5bw"
server_key: "ec06281eb4fbf32d5a457d3c61c80b6f"
server_key_iv: "0dbb5314152306884c1d55ecb8080c31"
server_key_version: "encryptionkey@0"
I20260928 23:50:31.828342 15979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:31.848263 15979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:31.848884 15979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:31.849063 15979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:31.855634 15979 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:46833
I20260928 23:50:31.855682 16046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:46833 every 8 connection(s)
I20260928 23:50:31.856552 15979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:50:31.859555 16047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:31.861910 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 15979
I20260928 23:50:31.862035 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:50:31.862310 6752 external_mini_cluster.cc:1428] Setting key c62c02349ed1d907706f57164be22145
I20260928 23:50:31.865342 16047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ca6bd2c1c6d4e0cad0e63d365f667c1: Bootstrap starting.
I20260928 23:50:31.867422 16047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ca6bd2c1c6d4e0cad0e63d365f667c1: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:31.868134 16047 log.cc:824] T 00000000000000000000000000000000 P 7ca6bd2c1c6d4e0cad0e63d365f667c1: Log is configured to *not* fsync() on all Append() calls
I20260928 23:50:31.870011 16047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ca6bd2c1c6d4e0cad0e63d365f667c1: No bootstrap required, opened a new log
I20260928 23:50:31.871062 16050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:31.863662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36888 (local address 127.6.152.62:46833)
0928 23:50:31.864257 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:31.864271 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:31.864303 (+ 32us) server_negotiation.cc:407] Connection header received
0928 23:50:31.864973 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:31.864990 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:31.865359 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:31.865710 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:31.866118 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:31.867053 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:31.867960 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:31.868462 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:31.868644 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:31.869477 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:31.869482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:31.869494 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:50:31.869863 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:31.869914 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:50:31.869932 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:31.870039 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:31.870183 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 23:50:31.870417 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":124,"threads_started":1}
I20260928 23:50:31.874100 16047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ca6bd2c1c6d4e0cad0e63d365f667c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ca6bd2c1c6d4e0cad0e63d365f667c1" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 46833 } }
I20260928 23:50:31.874329 16047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ca6bd2c1c6d4e0cad0e63d365f667c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:31.874413 16047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ca6bd2c1c6d4e0cad0e63d365f667c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ca6bd2c1c6d4e0cad0e63d365f667c1, State: Initialized, Role: FOLLOWER
I20260928 23:50:31.874845 16047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ca6bd2c1c6d4e0cad0e63d365f667c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ca6bd2c1c6d4e0cad0e63d365f667c1" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 46833 } }
I20260928 23:50:31.874997 16047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ca6bd2c1c6d4e0cad0e63d365f667c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:50:31.875077 16047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ca6bd2c1c6d4e0cad0e63d365f667c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:50:31.875172 16047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ca6bd2c1c6d4e0cad0e63d365f667c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:31.876368 16047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ca6bd2c1c6d4e0cad0e63d365f667c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ca6bd2c1c6d4e0cad0e63d365f667c1" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 46833 } }
I20260928 23:50:31.876766 16047 leader_election.cc:302] T 00000000000000000000000000000000 P 7ca6bd2c1c6d4e0cad0e63d365f667c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ca6bd2c1c6d4e0cad0e63d365f667c1; no voters:
I20260928 23:50:31.877134 16047 leader_election.cc:288] T 00000000000000000000000000000000 P 7ca6bd2c1c6d4e0cad0e63d365f667c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:50:31.877261 16052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ca6bd2c1c6d4e0cad0e63d365f667c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:50:31.877471 16052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ca6bd2c1c6d4e0cad0e63d365f667c1 [term 1 LEADER]: Becoming Leader. State: Replica: 7ca6bd2c1c6d4e0cad0e63d365f667c1, State: Running, Role: LEADER
I20260928 23:50:31.877782 16052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ca6bd2c1c6d4e0cad0e63d365f667c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ca6bd2c1c6d4e0cad0e63d365f667c1" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 46833 } }
I20260928 23:50:31.878487 16047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ca6bd2c1c6d4e0cad0e63d365f667c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:50:31.879428 16054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ca6bd2c1c6d4e0cad0e63d365f667c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ca6bd2c1c6d4e0cad0e63d365f667c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ca6bd2c1c6d4e0cad0e63d365f667c1" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 46833 } } }
I20260928 23:50:31.879664 16054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ca6bd2c1c6d4e0cad0e63d365f667c1 [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:31.880072 16061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:50:31.880288 16053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ca6bd2c1c6d4e0cad0e63d365f667c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ca6bd2c1c6d4e0cad0e63d365f667c1. Latest consensus state: current_term: 1 leader_uuid: "7ca6bd2c1c6d4e0cad0e63d365f667c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ca6bd2c1c6d4e0cad0e63d365f667c1" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 46833 } } }
I20260928 23:50:31.880414 16053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ca6bd2c1c6d4e0cad0e63d365f667c1 [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:31.883333 16061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:50:31.888679 16061 catalog_manager.cc:1347] Generated new cluster ID: 9eaa18f2e0e3404facaa3d686e3bbbcb
I20260928 23:50:31.888746 16061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:50:31.910583 16061 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:50:31.911733 16061 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:50:31.925226 16061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ca6bd2c1c6d4e0cad0e63d365f667c1: Generated new TSK 0
I20260928 23:50:31.926059 16061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:50:31.936458 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:46833
--builtin_ntp_servers=127.6.152.20:37803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 23:50:32.045246 16071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:32.045490 16071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:32.045526 16071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:32.049527 16071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:32.049573 16071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:32.049654 16071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:50:32.054346 16071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:37803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:46833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:32.055398 16071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:32.056910 16071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:32.066622 16076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:32.066645 16077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:32.066628 16079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:32.066955 16071 server_base.cc:1034] running on GCE node
I20260928 23:50:32.067327 16071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:32.067920 16071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:32.069089 16071 hybrid_clock.cc:648] HybridClock initialized: now 1790639432069068 us; error 34 us; skew 500 ppm
I20260928 23:50:32.070775 16071 webserver.cc:460] Webserver started at http://127.6.152.1:42427/ using document root <none> and password file <none>
I20260928 23:50:32.071334 16071 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:32.071419 16071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:32.071643 16071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:32.073745 16071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "06bc419cfa4f45fc9c880afa3f365369"
format_stamp: "Formatted at 2026-09-28 23:50:32 on dist-test-slave-p5bw"
server_key: "8bad5bf9e943c724eba2227b1b684425"
server_key_iv: "9c5f51cd19b7aa01c3f4e2dab555ae01"
server_key_version: "encryptionkey@0"
I20260928 23:50:32.074311 16071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "06bc419cfa4f45fc9c880afa3f365369"
format_stamp: "Formatted at 2026-09-28 23:50:32 on dist-test-slave-p5bw"
server_key: "8bad5bf9e943c724eba2227b1b684425"
server_key_iv: "9c5f51cd19b7aa01c3f4e2dab555ae01"
server_key_version: "encryptionkey@0"
I20260928 23:50:32.077950 16071 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260928 23:50:32.080652 16085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:32.081929 16071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 23:50:32.082067 16071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "06bc419cfa4f45fc9c880afa3f365369"
format_stamp: "Formatted at 2026-09-28 23:50:32 on dist-test-slave-p5bw"
server_key: "8bad5bf9e943c724eba2227b1b684425"
server_key_iv: "9c5f51cd19b7aa01c3f4e2dab555ae01"
server_key_version: "encryptionkey@0"
I20260928 23:50:32.082188 16071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:32.096267 16071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:32.097084 16071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:32.097280 16071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:32.097868 16071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:50:32.098801 16071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:32.098869 16071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:32.098938 16071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:32.098986 16071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:32.107398 16071 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:33501
I20260928 23:50:32.107460 16199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:33501 every 8 connection(s)
I20260928 23:50:32.108278 16071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:50:32.112867 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 16071
I20260928 23:50:32.112986 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
I20260928 23:50:32.113263 6752 external_mini_cluster.cc:1428] Setting key a18771d3c369ed0ec188085131426e0f
I20260928 23:50:32.115268 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:46833
--builtin_ntp_servers=127.6.152.20:37803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 23:50:32.118153 16050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:32.110040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:41451 (local address 127.6.152.62:46833)
0928 23:50:32.110172 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:32.110176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:32.111174 (+ 998us) server_negotiation.cc:407] Connection header received
0928 23:50:32.112485 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:32.112488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:32.112530 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:32.112651 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:32.113732 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:32.114441 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:32.116112 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:32.116341 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:32.116426 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:32.117616 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:32.117621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:32.117626 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:50:32.117641 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:32.117666 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:50:32.117681 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:32.117764 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:32.117887 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 23:50:32.117951 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:50:32.118863 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:32.110411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:46833 (local address 127.6.152.1:41451)
0928 23:50:32.111029 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:32.111068 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:32.112310 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:32.112793 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:32.112800 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:32.112925 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:32.113555 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:32.113587 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:32.114568 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:32.114571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:32.115987 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:32.116002 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:32.116496 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:32.116503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:32.116718 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:32.117370 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:50:32.117371 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:32.117409 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:50:32.117469 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:32.117763 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:32.117769 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:32.117773 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:50:32.117863 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 23:50:32.118068 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":105,"threads_started":1}
I20260928 23:50:32.120018 16200 heartbeater.cc:347] Connected to a master server at 127.6.152.62:46833
I20260928 23:50:32.120242 16200 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:32.120716 16200 heartbeater.cc:511] Master 127.6.152.62:46833 requested a full tablet report, sending...
I20260928 23:50:32.122423 16011 ts_manager.cc:194] Registered new tserver with Master: 06bc419cfa4f45fc9c880afa3f365369 (127.6.152.1:33501)
I20260928 23:50:32.124197 16011 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.152.1:41451
W20260928 23:50:32.222972 16203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:32.223377 16203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:32.223450 16203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:32.227384 16203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:32.227440 16203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:32.227494 16203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:50:32.231987 16203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:37803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:46833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:32.232950 16203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:32.234236 16203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:32.243158 16209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:32.243243 16211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:32.243148 16208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:32.243535 16203 server_base.cc:1034] running on GCE node
I20260928 23:50:32.243897 16203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:32.244520 16203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:32.245692 16203 hybrid_clock.cc:648] HybridClock initialized: now 1790639432245644 us; error 67 us; skew 500 ppm
I20260928 23:50:32.247473 16203 webserver.cc:460] Webserver started at http://127.6.152.2:43885/ using document root <none> and password file <none>
I20260928 23:50:32.248039 16203 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:32.248126 16203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:32.248347 16203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:32.250372 16203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "6e9219618b1f4dc988646db603b8726a"
format_stamp: "Formatted at 2026-09-28 23:50:32 on dist-test-slave-p5bw"
server_key: "c953775acb3ee02ff321c7c6d3adf16e"
server_key_iv: "1bf59273a6f67c02cdbd562e029d8664"
server_key_version: "encryptionkey@0"
I20260928 23:50:32.250917 16203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "6e9219618b1f4dc988646db603b8726a"
format_stamp: "Formatted at 2026-09-28 23:50:32 on dist-test-slave-p5bw"
server_key: "c953775acb3ee02ff321c7c6d3adf16e"
server_key_iv: "1bf59273a6f67c02cdbd562e029d8664"
server_key_version: "encryptionkey@0"
I20260928 23:50:32.254415 16203 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260928 23:50:32.256788 16217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:32.258026 16203 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 23:50:32.258145 16203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "6e9219618b1f4dc988646db603b8726a"
format_stamp: "Formatted at 2026-09-28 23:50:32 on dist-test-slave-p5bw"
server_key: "c953775acb3ee02ff321c7c6d3adf16e"
server_key_iv: "1bf59273a6f67c02cdbd562e029d8664"
server_key_version: "encryptionkey@0"
I20260928 23:50:32.258251 16203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:32.276757 16203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:32.277441 16203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:32.277632 16203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:32.278190 16203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:50:32.279176 16203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:32.279242 16203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:32.279306 16203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:32.279353 16203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:32.288334 16203 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:44751
I20260928 23:50:32.288358 16331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:44751 every 8 connection(s)
I20260928 23:50:32.289422 16203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:50:32.291395 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 16203
I20260928 23:50:32.291499 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:50:32.291806 6752 external_mini_cluster.cc:1428] Setting key e3795d70e114ca05d90bedecf987db44
I20260928 23:50:32.293874 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:46833
--builtin_ntp_servers=127.6.152.20:37803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 23:50:32.299962 16050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:32.291248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:42219 (local address 127.6.152.62:46833)
0928 23:50:32.291384 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:32.291392 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:32.292724 (+ 1332us) server_negotiation.cc:407] Connection header received
0928 23:50:32.294400 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:32.294406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:32.294473 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:32.294556 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:32.295578 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:32.296096 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:32.297901 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:32.298118 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:32.298200 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:32.299504 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:32.299507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:32.299509 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:50:32.299521 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:32.299540 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:50:32.299547 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:32.299605 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:32.299804 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 23:50:32.299852 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 23:50:32.301270 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:32.291771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:46833 (local address 127.6.152.2:42219)
0928 23:50:32.292543 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:32.292608 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:32.294197 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:32.294690 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:32.294697 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:32.294781 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:32.295424 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:32.295446 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:32.296220 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:32.296226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:32.297764 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:32.297778 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:32.298277 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:32.298285 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:32.298526 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:32.299183 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:50:32.299187 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:32.299249 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:50:32.299339 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:32.299633 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:32.299645 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:32.299654 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 23:50:32.299796 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 23:50:32.300131 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":127,"threads_started":1}
I20260928 23:50:32.302469 16332 heartbeater.cc:347] Connected to a master server at 127.6.152.62:46833
I20260928 23:50:32.302706 16332 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:32.303188 16332 heartbeater.cc:511] Master 127.6.152.62:46833 requested a full tablet report, sending...
I20260928 23:50:32.304168 16011 ts_manager.cc:194] Registered new tserver with Master: 6e9219618b1f4dc988646db603b8726a (127.6.152.2:44751)
I20260928 23:50:32.304718 16011 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.152.2:42219
W20260928 23:50:32.404613 16335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:32.404901 16335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:32.404966 16335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:32.408877 16335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:32.408951 16335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:32.409039 16335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:50:32.413643 16335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:37803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:46833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:32.414661 16335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:32.415951 16335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:32.425113 16343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:32.425148 16341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:32.425135 16340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:32.425525 16335 server_base.cc:1034] running on GCE node
I20260928 23:50:32.425837 16335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:32.426420 16335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:32.427591 16335 hybrid_clock.cc:648] HybridClock initialized: now 1790639432427582 us; error 36 us; skew 500 ppm
I20260928 23:50:32.429530 16335 webserver.cc:460] Webserver started at http://127.6.152.3:42205/ using document root <none> and password file <none>
I20260928 23:50:32.430092 16335 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:32.430153 16335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:32.430389 16335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:32.432329 16335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "9cb6027a20aa44d48b1d77171ae0eb30"
format_stamp: "Formatted at 2026-09-28 23:50:32 on dist-test-slave-p5bw"
server_key: "3f21598bf6316f4476b94babb7369a14"
server_key_iv: "acaca895bfb9e1e9d61db2a6fac7695d"
server_key_version: "encryptionkey@0"
I20260928 23:50:32.432911 16335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "9cb6027a20aa44d48b1d77171ae0eb30"
format_stamp: "Formatted at 2026-09-28 23:50:32 on dist-test-slave-p5bw"
server_key: "3f21598bf6316f4476b94babb7369a14"
server_key_iv: "acaca895bfb9e1e9d61db2a6fac7695d"
server_key_version: "encryptionkey@0"
I20260928 23:50:32.436347 16335 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 23:50:32.438732 16349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:32.439930 16335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:50:32.440045 16335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "9cb6027a20aa44d48b1d77171ae0eb30"
format_stamp: "Formatted at 2026-09-28 23:50:32 on dist-test-slave-p5bw"
server_key: "3f21598bf6316f4476b94babb7369a14"
server_key_iv: "acaca895bfb9e1e9d61db2a6fac7695d"
server_key_version: "encryptionkey@0"
I20260928 23:50:32.440140 16335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:32.456735 16335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:32.457456 16335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:32.457635 16335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:32.458230 16335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:50:32.459098 16335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:32.459143 16335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:32.459205 16335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:32.459240 16335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:32.467711 16335 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:40483
I20260928 23:50:32.467742 16463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:40483 every 8 connection(s)
I20260928 23:50:32.468564 16335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:50:32.470191 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 16335
I20260928 23:50:32.470293 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:50:32.470607 6752 external_mini_cluster.cc:1428] Setting key 150b73a1dc1b456e5c9361819d1cb03e
I20260928 23:50:32.477355 16050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:32.470376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:37459 (local address 127.6.152.62:46833)
0928 23:50:32.470514 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:32.470518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:32.471713 (+ 1195us) server_negotiation.cc:407] Connection header received
0928 23:50:32.473210 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:32.473212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:32.473257 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:32.473327 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:32.474150 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:32.474660 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:32.475717 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:32.475927 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:32.475997 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:32.476963 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:32.476966 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:32.476968 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:50:32.476979 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:32.476998 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:50:32.477005 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:32.477061 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:32.477188 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 23:50:32.477236 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:50:32.478155 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:32.470861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:46833 (local address 127.6.152.3:37459)
0928 23:50:32.471569 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:32.471618 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:32.473029 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:32.473471 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:32.473477 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:32.473576 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:32.474038 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:32.474050 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:32.474759 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:32.474761 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:32.475618 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:32.475627 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:32.476038 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:32.476040 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:32.476175 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:32.476693 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:50:32.476694 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:32.476732 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:50:32.476791 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:32.477072 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:32.477078 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:32.477082 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:50:32.477174 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 23:50:32.477391 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":135,"threads_started":1}
I20260928 23:50:32.479205 16464 heartbeater.cc:347] Connected to a master server at 127.6.152.62:46833
I20260928 23:50:32.479432 16464 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:32.479864 16464 heartbeater.cc:511] Master 127.6.152.62:46833 requested a full tablet report, sending...
I20260928 23:50:32.480772 16011 ts_manager.cc:194] Registered new tserver with Master: 9cb6027a20aa44d48b1d77171ae0eb30 (127.6.152.3:40483)
I20260928 23:50:32.481362 16011 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.152.3:37459
I20260928 23:50:32.484354 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 23:50:32.487180 16050 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:32.486235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36898 (local address 127.6.152.62:46833)
0928 23:50:32.486344 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:32.486347 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:32.486537 (+ 190us) server_negotiation.cc:407] Connection header received
0928 23:50:32.486664 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:32.486668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:32.486715 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:32.486794 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:32.487056 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36898: BlockingRecv error: recv got EOF from 127.0.0.1:36898 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:50:32.487334 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 16071
I20260928 23:50:32.493180 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 16203
I20260928 23:50:32.498327 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 16335
I20260928 23:50:32.503387 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 15979
2026-09-28T23:50:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (845 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T23:50:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:50:32Z Disabled control of system clock
I20260928 23:50:32.531541 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:42089
--webserver_interface=127.6.152.62
--webserver_port=0
--builtin_ntp_servers=127.6.152.20:33863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:42089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 23:50:32.637847 16475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:32.638090 16475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:32.638126 16475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:32.641965 16475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:50:32.642014 16475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:32.642028 16475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:32.642045 16475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:50:32.642060 16475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:50:32.646669 16475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:33863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:42089
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:42089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:32.647611 16475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:32.648895 16475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:50:32.657218 16475 server_base.cc:1034] running on GCE node
W20260928 23:50:32.657107 16484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:32.657107 16480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:32.657119 16482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:32.657935 16475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:32.659001 16475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:32.660176 16475 hybrid_clock.cc:648] HybridClock initialized: now 1790639432660145 us; error 44 us; skew 500 ppm
I20260928 23:50:32.661870 16475 webserver.cc:460] Webserver started at http://127.6.152.62:38163/ using document root <none> and password file <none>
I20260928 23:50:32.662427 16475 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:32.662509 16475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:32.662729 16475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:32.664659 16475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "3b592b015b5340ddb0bc93666b07155a"
format_stamp: "Formatted at 2026-09-28 23:50:32 on dist-test-slave-p5bw"
server_key: "0e2f3535a2a7aadc7e15fa9c66affb5c"
server_key_iv: "f26d989dff6e005da102a98d95d9a713"
server_key_version: "encryptionkey@0"
I20260928 23:50:32.665252 16475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "3b592b015b5340ddb0bc93666b07155a"
format_stamp: "Formatted at 2026-09-28 23:50:32 on dist-test-slave-p5bw"
server_key: "0e2f3535a2a7aadc7e15fa9c66affb5c"
server_key_iv: "f26d989dff6e005da102a98d95d9a713"
server_key_version: "encryptionkey@0"
I20260928 23:50:32.668653 16475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:50:32.670904 16490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:32.672006 16475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:50:32.672138 16475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "3b592b015b5340ddb0bc93666b07155a"
format_stamp: "Formatted at 2026-09-28 23:50:32 on dist-test-slave-p5bw"
server_key: "0e2f3535a2a7aadc7e15fa9c66affb5c"
server_key_iv: "f26d989dff6e005da102a98d95d9a713"
server_key_version: "encryptionkey@0"
I20260928 23:50:32.672241 16475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:32.676599 16475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:32.677202 16475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:32.677377 16475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:32.683771 16475 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:42089
I20260928 23:50:32.683779 16542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:42089 every 8 connection(s)
I20260928 23:50:32.684640 16475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:50:32.686518 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 16475
I20260928 23:50:32.686630 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:50:32.686905 6752 external_mini_cluster.cc:1428] Setting key 24051f1f888d80f6543fd0b64c85d176
I20260928 23:50:32.687958 16543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:32.692116 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:32.688224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55134 (local address 127.6.152.62:42089)
0928 23:50:32.688805 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:32.688865 (+ 60us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:32.688901 (+ 36us) server_negotiation.cc:407] Connection header received
0928 23:50:32.689525 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:32.689540 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:32.689866 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:32.690138 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:32.690933 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:32.690937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:32.690947 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:50:32.690957 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:32.691007 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:50:32.691022 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:32.691134 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:32.691235 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 23:50:32.691482 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":129,"threads_started":1}
I20260928 23:50:32.694205 16543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b592b015b5340ddb0bc93666b07155a: Bootstrap starting.
I20260928 23:50:32.696175 16543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b592b015b5340ddb0bc93666b07155a: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:32.696954 16543 log.cc:824] T 00000000000000000000000000000000 P 3b592b015b5340ddb0bc93666b07155a: Log is configured to *not* fsync() on all Append() calls
I20260928 23:50:32.698637 16543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b592b015b5340ddb0bc93666b07155a: No bootstrap required, opened a new log
I20260928 23:50:32.702025 16543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b592b015b5340ddb0bc93666b07155a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b592b015b5340ddb0bc93666b07155a" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 42089 } }
I20260928 23:50:32.702243 16543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b592b015b5340ddb0bc93666b07155a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:32.702315 16543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b592b015b5340ddb0bc93666b07155a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b592b015b5340ddb0bc93666b07155a, State: Initialized, Role: FOLLOWER
I20260928 23:50:32.702710 16543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b592b015b5340ddb0bc93666b07155a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b592b015b5340ddb0bc93666b07155a" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 42089 } }
I20260928 23:50:32.702850 16543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b592b015b5340ddb0bc93666b07155a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:50:32.702942 16543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b592b015b5340ddb0bc93666b07155a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:50:32.703056 16543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b592b015b5340ddb0bc93666b07155a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:32.703962 16543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b592b015b5340ddb0bc93666b07155a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b592b015b5340ddb0bc93666b07155a" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 42089 } }
I20260928 23:50:32.704275 16543 leader_election.cc:302] T 00000000000000000000000000000000 P 3b592b015b5340ddb0bc93666b07155a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b592b015b5340ddb0bc93666b07155a; no voters:
I20260928 23:50:32.704553 16543 leader_election.cc:288] T 00000000000000000000000000000000 P 3b592b015b5340ddb0bc93666b07155a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:50:32.704664 16548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b592b015b5340ddb0bc93666b07155a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:50:32.704893 16548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b592b015b5340ddb0bc93666b07155a [term 1 LEADER]: Becoming Leader. State: Replica: 3b592b015b5340ddb0bc93666b07155a, State: Running, Role: LEADER
I20260928 23:50:32.705214 16548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b592b015b5340ddb0bc93666b07155a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b592b015b5340ddb0bc93666b07155a" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 42089 } }
I20260928 23:50:32.705701 16543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b592b015b5340ddb0bc93666b07155a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:50:32.706765 16550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b592b015b5340ddb0bc93666b07155a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b592b015b5340ddb0bc93666b07155a. Latest consensus state: current_term: 1 leader_uuid: "3b592b015b5340ddb0bc93666b07155a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b592b015b5340ddb0bc93666b07155a" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 42089 } } }
I20260928 23:50:32.706887 16550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b592b015b5340ddb0bc93666b07155a [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:32.707168 16549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b592b015b5340ddb0bc93666b07155a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b592b015b5340ddb0bc93666b07155a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b592b015b5340ddb0bc93666b07155a" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 42089 } } }
I20260928 23:50:32.707333 16549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b592b015b5340ddb0bc93666b07155a [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:32.707271 16557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:50:32.710934 16557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:50:32.715909 16557 catalog_manager.cc:1347] Generated new cluster ID: b40ad4fe1cff4158aa9a4c1c9de7b4b0
I20260928 23:50:32.715974 16557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:50:32.730346 16557 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:50:32.730657 16557 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:50:32.736128 16557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b592b015b5340ddb0bc93666b07155a: Generated new TSK 0
I20260928 23:50:32.736749 16557 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:50:32.745160 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:42089
--builtin_ntp_servers=127.6.152.20:33863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 23:50:32.854178 16567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:32.854411 16567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:32.854486 16567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:32.858630 16567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:32.858680 16567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:32.858772 16567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:50:32.863582 16567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:33863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:42089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:32.864614 16567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:32.865978 16567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:32.875391 16576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:32.875491 16567 server_base.cc:1034] running on GCE node
W20260928 23:50:32.875465 16573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:32.875398 16574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:32.876118 16567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:32.876756 16567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:32.877955 16567 hybrid_clock.cc:648] HybridClock initialized: now 1790639432877931 us; error 39 us; skew 500 ppm
I20260928 23:50:32.879810 16567 webserver.cc:460] Webserver started at http://127.6.152.1:35273/ using document root <none> and password file <none>
I20260928 23:50:32.880374 16567 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:32.880470 16567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:32.880720 16567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:32.882777 16567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "4504f207ed084933bb8cd59b37981dd7"
format_stamp: "Formatted at 2026-09-28 23:50:32 on dist-test-slave-p5bw"
server_key: "6865370cdf483fcdfc77bd55fe96f1c7"
server_key_iv: "486a67e43d78e677993ebbd8b5404e73"
server_key_version: "encryptionkey@0"
I20260928 23:50:32.883344 16567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "4504f207ed084933bb8cd59b37981dd7"
format_stamp: "Formatted at 2026-09-28 23:50:32 on dist-test-slave-p5bw"
server_key: "6865370cdf483fcdfc77bd55fe96f1c7"
server_key_iv: "486a67e43d78e677993ebbd8b5404e73"
server_key_version: "encryptionkey@0"
I20260928 23:50:32.886952 16567 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 23:50:32.889422 16582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:32.890563 16567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:50:32.890688 16567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "4504f207ed084933bb8cd59b37981dd7"
format_stamp: "Formatted at 2026-09-28 23:50:32 on dist-test-slave-p5bw"
server_key: "6865370cdf483fcdfc77bd55fe96f1c7"
server_key_iv: "486a67e43d78e677993ebbd8b5404e73"
server_key_version: "encryptionkey@0"
I20260928 23:50:32.890796 16567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:32.895217 16567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:32.895824 16567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:32.895999 16567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:32.896500 16567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:50:32.897400 16567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:32.897467 16567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:32.897536 16567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:32.897581 16567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:32.905843 16567 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:33797
I20260928 23:50:32.905866 16695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:33797 every 8 connection(s)
I20260928 23:50:32.906720 16567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:50:32.910966 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 16567
I20260928 23:50:32.911089 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
I20260928 23:50:32.911429 6752 external_mini_cluster.cc:1428] Setting key 424f1d26f56215e7d65d977fd4bcdbed
I20260928 23:50:32.912886 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:32.908532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:59903 (local address 127.6.152.62:42089)
0928 23:50:32.908687 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:32.908692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:32.909828 (+ 1136us) server_negotiation.cc:407] Connection header received
0928 23:50:32.911141 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:32.911145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:32.911190 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:32.911299 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:32.912374 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:32.912377 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:32.912380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:50:32.912397 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:32.912416 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:50:32.912425 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:32.912519 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:32.912678 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 23:50:32.912722 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 23:50:32.913412 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:42089
--builtin_ntp_servers=127.6.152.20:33863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 23:50:32.913787 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:32.909038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:42089 (local address 127.6.152.1:59903)
0928 23:50:32.909680 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:32.909721 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:32.910904 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:32.911435 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:32.911441 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:32.911571 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:32.912127 (+ 556us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:50:32.912129 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:32.912166 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:50:32.912232 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:32.912510 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:32.912519 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:32.912525 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 23:50:32.912704 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 23:50:32.912970 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":109,"threads_started":1}
I20260928 23:50:32.915071 16696 heartbeater.cc:347] Connected to a master server at 127.6.152.62:42089
I20260928 23:50:32.915289 16696 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:32.915664 16696 heartbeater.cc:511] Master 127.6.152.62:42089 requested a full tablet report, sending...
I20260928 23:50:32.917133 16507 ts_manager.cc:194] Registered new tserver with Master: 4504f207ed084933bb8cd59b37981dd7 (127.6.152.1:33797)
W20260928 23:50:33.022311 16699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:33.022576 16699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:33.022617 16699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:33.026655 16699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:33.026710 16699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:33.026798 16699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:50:33.031620 16699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:33863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:42089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:33.032744 16699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:33.034171 16699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:33.043594 16708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:33.043579 16705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:33.043579 16704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:33.043635 16699 server_base.cc:1034] running on GCE node
I20260928 23:50:33.044173 16699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:33.044764 16699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:33.045957 16699 hybrid_clock.cc:648] HybridClock initialized: now 1790639433045942 us; error 26 us; skew 500 ppm
I20260928 23:50:33.047662 16699 webserver.cc:460] Webserver started at http://127.6.152.2:37235/ using document root <none> and password file <none>
I20260928 23:50:33.048223 16699 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:33.048283 16699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:33.048494 16699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:33.050556 16699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "307ad0909b9b4b90b00a56135b7a5d44"
format_stamp: "Formatted at 2026-09-28 23:50:33 on dist-test-slave-p5bw"
server_key: "1a039726fdb039d567ade70c5580b9c1"
server_key_iv: "19ccb09ea65ab446da532bf27d5b082f"
server_key_version: "encryptionkey@0"
I20260928 23:50:33.051106 16699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "307ad0909b9b4b90b00a56135b7a5d44"
format_stamp: "Formatted at 2026-09-28 23:50:33 on dist-test-slave-p5bw"
server_key: "1a039726fdb039d567ade70c5580b9c1"
server_key_iv: "19ccb09ea65ab446da532bf27d5b082f"
server_key_version: "encryptionkey@0"
I20260928 23:50:33.054742 16699 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 23:50:33.057268 16714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:33.058658 16699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:50:33.058797 16699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "307ad0909b9b4b90b00a56135b7a5d44"
format_stamp: "Formatted at 2026-09-28 23:50:33 on dist-test-slave-p5bw"
server_key: "1a039726fdb039d567ade70c5580b9c1"
server_key_iv: "19ccb09ea65ab446da532bf27d5b082f"
server_key_version: "encryptionkey@0"
I20260928 23:50:33.058917 16699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:33.063377 16699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:33.063944 16699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:33.064118 16699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:33.064620 16699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:50:33.065500 16699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:33.065567 16699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:33.065629 16699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:33.065673 16699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:33.074232 16699 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:37781
I20260928 23:50:33.074252 16827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:37781 every 8 connection(s)
I20260928 23:50:33.075126 16699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:50:33.079596 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 16699
I20260928 23:50:33.079725 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:50:33.080046 6752 external_mini_cluster.cc:1428] Setting key 3029bd0cd79a13ff4d87cd267faa93eb
I20260928 23:50:33.081187 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:33.076882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:51171 (local address 127.6.152.62:42089)
0928 23:50:33.077032 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:33.077036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:33.078120 (+ 1084us) server_negotiation.cc:407] Connection header received
0928 23:50:33.079378 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:33.079380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:33.079416 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:33.079491 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:33.080686 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:33.080689 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:33.080692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:50:33.080704 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:33.080725 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:50:33.080734 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:33.080848 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:33.081004 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 23:50:33.081048 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:50:33.082199 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:42089
--builtin_ntp_servers=127.6.152.20:33863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 23:50:33.082312 16830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:33.077360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:42089 (local address 127.6.152.2:51171)
0928 23:50:33.077974 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:33.078012 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:33.079162 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:33.079683 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:33.079694 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:33.079809 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:33.080402 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:50:33.080404 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:33.080457 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:50:33.080539 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:33.080853 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:33.080864 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:33.080872 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 23:50:33.081046 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 23:50:33.081394 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":104,"threads_started":1}
I20260928 23:50:33.083374 16828 heartbeater.cc:347] Connected to a master server at 127.6.152.62:42089
I20260928 23:50:33.083596 16828 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:33.083992 16828 heartbeater.cc:511] Master 127.6.152.62:42089 requested a full tablet report, sending...
I20260928 23:50:33.084997 16507 ts_manager.cc:194] Registered new tserver with Master: 307ad0909b9b4b90b00a56135b7a5d44 (127.6.152.2:37781)
W20260928 23:50:33.192067 16831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:33.192333 16831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:33.192399 16831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:33.196708 16831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:33.196771 16831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:33.196893 16831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:50:33.201928 16831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:33863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:42089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:33.202953 16831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:33.204654 16831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:33.214797 16836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:33.214954 16831 server_base.cc:1034] running on GCE node
W20260928 23:50:33.214777 16837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:33.214821 16839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:33.215507 16831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:33.216132 16831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:33.217312 16831 hybrid_clock.cc:648] HybridClock initialized: now 1790639433217282 us; error 44 us; skew 500 ppm
I20260928 23:50:33.219142 16831 webserver.cc:460] Webserver started at http://127.6.152.3:43961/ using document root <none> and password file <none>
I20260928 23:50:33.219913 16831 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:33.220091 16831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:33.220333 16831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:33.222373 16831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "0038ecd35c9c42fc8b7636bfc36ff50a"
format_stamp: "Formatted at 2026-09-28 23:50:33 on dist-test-slave-p5bw"
server_key: "d8ff16e233b297bd891ef8850d34a2ce"
server_key_iv: "8ab9fdc948853a715669a448bcdffa97"
server_key_version: "encryptionkey@0"
I20260928 23:50:33.222996 16831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "0038ecd35c9c42fc8b7636bfc36ff50a"
format_stamp: "Formatted at 2026-09-28 23:50:33 on dist-test-slave-p5bw"
server_key: "d8ff16e233b297bd891ef8850d34a2ce"
server_key_iv: "8ab9fdc948853a715669a448bcdffa97"
server_key_version: "encryptionkey@0"
I20260928 23:50:33.226787 16831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:50:33.229379 16846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:33.230650 16831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:50:33.230806 16831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "0038ecd35c9c42fc8b7636bfc36ff50a"
format_stamp: "Formatted at 2026-09-28 23:50:33 on dist-test-slave-p5bw"
server_key: "d8ff16e233b297bd891ef8850d34a2ce"
server_key_iv: "8ab9fdc948853a715669a448bcdffa97"
server_key_version: "encryptionkey@0"
I20260928 23:50:33.230938 16831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:33.235808 16831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:33.236474 16831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:33.236716 16831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:33.237288 16831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:50:33.238361 16831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:33.238495 16831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:33.238548 16831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:33.238570 16831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:33.246598 16831 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:38269
I20260928 23:50:33.246628 16959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:38269 every 8 connection(s)
I20260928 23:50:33.247460 16831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:50:33.248327 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 16831
I20260928 23:50:33.248437 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:50:33.248730 6752 external_mini_cluster.cc:1428] Setting key f2d53cc81998bd97a334d2af271e88e4
I20260928 23:50:33.253734 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:33.249383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:48435 (local address 127.6.152.62:42089)
0928 23:50:33.249537 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:33.249542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:33.250817 (+ 1275us) server_negotiation.cc:407] Connection header received
0928 23:50:33.252223 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:33.252225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:33.252260 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:33.252324 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:33.253395 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:33.253397 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:33.253399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:50:33.253409 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:33.253427 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:50:33.253434 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:33.253500 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:33.253595 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 23:50:33.253638 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:50:33.254576 16962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:33.249867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:42089 (local address 127.6.152.3:48435)
0928 23:50:33.250671 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:33.250720 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:33.252037 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:33.252469 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:33.252475 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:33.252562 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:33.253168 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:50:33.253170 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:33.253209 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:50:33.253275 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:33.253491 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:33.253495 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:33.253499 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:50:33.253615 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 23:50:33.253858 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":198,"threads_started":1}
I20260928 23:50:33.255633 16960 heartbeater.cc:347] Connected to a master server at 127.6.152.62:42089
I20260928 23:50:33.255847 16960 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:33.256223 16960 heartbeater.cc:511] Master 127.6.152.62:42089 requested a full tablet report, sending...
I20260928 23:50:33.257185 16507 ts_manager.cc:194] Registered new tserver with Master: 0038ecd35c9c42fc8b7636bfc36ff50a (127.6.152.3:38269)
I20260928 23:50:33.262324 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 23:50:33.265173 16546 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:50:33.264265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55146 (local address 127.6.152.62:42089)
0928 23:50:33.264412 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:33.264416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:33.264616 (+ 200us) server_negotiation.cc:407] Connection header received
0928 23:50:33.264698 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:33.264700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:33.264727 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:33.264788 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:33.265058 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55146: BlockingRecv error: recv got EOF from 127.0.0.1:55146 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:50:33.265295 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 16567
I20260928 23:50:33.271281 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 16699
I20260928 23:50:33.276741 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 16831
I20260928 23:50:33.282138 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 16475
2026-09-28T23:50:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (778 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:50:33 dist-test-slave-p5bw krb5kdc[16970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:50:33 dist-test-slave-p5bw krb5kdc[16970](info): set up 2 sockets
Sep 28 23:50:33 dist-test-slave-p5bw krb5kdc[16970](info): commencing operation
krb5kdc: starting...
W20260928 23:50:34.326536 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:50:34 dist-test-slave-p5bw krb5kdc[16970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:50:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:50:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:50:34.447597 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:42953
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:42953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:50:34.554250 16983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:34.554481 16983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:34.554522 16983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:34.558349 16983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:50:34.558393 16983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:34.558413 16983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:34.558429 16983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:50:34.558444 16983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:50:34.563531 16983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:42953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:42953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:34.564443 16983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:34.565376 16983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:34.573570 16989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:34.573570 16988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:34.573577 16991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:34.573969 16983 server_base.cc:1034] running on GCE node
I20260928 23:50:34.574481 16983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:34.575502 16983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:34.576682 16983 hybrid_clock.cc:648] HybridClock initialized: now 1790639434576659 us; error 40 us; skew 500 ppm
Sep 28 23:50:34 dist-test-slave-p5bw krb5kdc[16970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639434, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:34.579303 16983 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:50:34.580160 16983 webserver.cc:460] Webserver started at http://127.6.152.62:46681/ using document root <none> and password file <none>
I20260928 23:50:34.580691 16983 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:34.580780 16983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:34.581030 16983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:34.583021 16983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "ad2feae16b184a7db3fbfccc28e3b2b3"
format_stamp: "Formatted at 2026-09-28 23:50:34 on dist-test-slave-p5bw"
server_key: "3cded57777ca5e21fab083827a4336a2"
server_key_iv: "d21f344647fd4d0f2852df7409ed8d8b"
server_key_version: "encryptionkey@0"
I20260928 23:50:34.583575 16983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "ad2feae16b184a7db3fbfccc28e3b2b3"
format_stamp: "Formatted at 2026-09-28 23:50:34 on dist-test-slave-p5bw"
server_key: "3cded57777ca5e21fab083827a4336a2"
server_key_iv: "d21f344647fd4d0f2852df7409ed8d8b"
server_key_version: "encryptionkey@0"
I20260928 23:50:34.587224 16983 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 23:50:34.589597 16998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:34.590749 16983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:50:34.590885 16983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "ad2feae16b184a7db3fbfccc28e3b2b3"
format_stamp: "Formatted at 2026-09-28 23:50:34 on dist-test-slave-p5bw"
server_key: "3cded57777ca5e21fab083827a4336a2"
server_key_iv: "d21f344647fd4d0f2852df7409ed8d8b"
server_key_version: "encryptionkey@0"
I20260928 23:50:34.590996 16983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:34.608018 16983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:34.610847 16983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:34.611043 16983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:34.617784 16983 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:42953
I20260928 23:50:34.618058 17050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:42953 every 8 connection(s)
I20260928 23:50:34.618774 16983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:50:34.621385 17051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:34.623485 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 16983
I20260928 23:50:34.623610 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:50:34.623908 6752 external_mini_cluster.cc:1428] Setting key 16f4ff5d5de0740bd09aa9a850691c88
I20260928 23:50:34.627465 17051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad2feae16b184a7db3fbfccc28e3b2b3: Bootstrap starting.
I20260928 23:50:34.629930 17051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad2feae16b184a7db3fbfccc28e3b2b3: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:34.630710 17051 log.cc:824] T 00000000000000000000000000000000 P ad2feae16b184a7db3fbfccc28e3b2b3: Log is configured to *not* fsync() on all Append() calls
I20260928 23:50:34.632560 17051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad2feae16b184a7db3fbfccc28e3b2b3: No bootstrap required, opened a new log
Sep 28 23:50:34 dist-test-slave-p5bw krb5kdc[16970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:50:34.636909 17051 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad2feae16b184a7db3fbfccc28e3b2b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2feae16b184a7db3fbfccc28e3b2b3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 42953 } }
I20260928 23:50:34.637166 17051 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad2feae16b184a7db3fbfccc28e3b2b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:34.637224 17051 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad2feae16b184a7db3fbfccc28e3b2b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2feae16b184a7db3fbfccc28e3b2b3, State: Initialized, Role: FOLLOWER
I20260928 23:50:34.637718 17051 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad2feae16b184a7db3fbfccc28e3b2b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2feae16b184a7db3fbfccc28e3b2b3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 42953 } }
I20260928 23:50:34.637874 17051 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad2feae16b184a7db3fbfccc28e3b2b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:50:34.637953 17051 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad2feae16b184a7db3fbfccc28e3b2b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:50:34.638057 17051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad2feae16b184a7db3fbfccc28e3b2b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:34.639109 17051 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad2feae16b184a7db3fbfccc28e3b2b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2feae16b184a7db3fbfccc28e3b2b3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 42953 } }
I20260928 23:50:34.639258 17054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:34.625313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55454 (local address 127.6.152.62:42953)
0928 23:50:34.625710 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:34.625725 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:34.625759 (+ 34us) server_negotiation.cc:407] Connection header received
0928 23:50:34.626502 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:34.626521 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:34.626844 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:34.627162 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:34.628077 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:34.629140 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:34.630278 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:34.630724 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:34.630890 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:34.633366 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:34.633373 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:34.633387 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:34.633417 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:34.635986 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:34.636397 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:34.636402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:34.636408 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:34.636540 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:34.636796 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:34.636799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:34.636801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:34.637328 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:34.637535 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:34.637811 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 23:50:34.638296 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":118,"threads_started":1}
I20260928 23:50:34.639472 17051 leader_election.cc:302] T 00000000000000000000000000000000 P ad2feae16b184a7db3fbfccc28e3b2b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad2feae16b184a7db3fbfccc28e3b2b3; no voters:
I20260928 23:50:34.639778 17051 leader_election.cc:288] T 00000000000000000000000000000000 P ad2feae16b184a7db3fbfccc28e3b2b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:50:34.639884 17056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad2feae16b184a7db3fbfccc28e3b2b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:50:34.640080 17056 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad2feae16b184a7db3fbfccc28e3b2b3 [term 1 LEADER]: Becoming Leader. State: Replica: ad2feae16b184a7db3fbfccc28e3b2b3, State: Running, Role: LEADER
I20260928 23:50:34.640930 17051 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad2feae16b184a7db3fbfccc28e3b2b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:50:34.641443 17056 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad2feae16b184a7db3fbfccc28e3b2b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2feae16b184a7db3fbfccc28e3b2b3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 42953 } }
I20260928 23:50:34.643078 17058 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad2feae16b184a7db3fbfccc28e3b2b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad2feae16b184a7db3fbfccc28e3b2b3. Latest consensus state: current_term: 1 leader_uuid: "ad2feae16b184a7db3fbfccc28e3b2b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2feae16b184a7db3fbfccc28e3b2b3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 42953 } } }
I20260928 23:50:34.643126 17057 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad2feae16b184a7db3fbfccc28e3b2b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad2feae16b184a7db3fbfccc28e3b2b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2feae16b184a7db3fbfccc28e3b2b3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 42953 } } }
I20260928 23:50:34.643505 17058 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad2feae16b184a7db3fbfccc28e3b2b3 [sys.catalog]: This master's current role is: LEADER
I20260928 23:50:34.643512 17057 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad2feae16b184a7db3fbfccc28e3b2b3 [sys.catalog]: This master's current role is: LEADER
W20260928 23:50:34.645659 17070 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad2feae16b184a7db3fbfccc28e3b2b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:50:34.645733 17070 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:50:34.645804 17071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:50:34.646626 17071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:50:34.651686 17071 catalog_manager.cc:1347] Generated new cluster ID: 6a6446529cd24df18c6005c74275fdde
I20260928 23:50:34.651749 17071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:50:34.664994 17071 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:50:34.665956 17071 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:50:34.678962 17071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad2feae16b184a7db3fbfccc28e3b2b3: Generated new TSK 0
I20260928 23:50:34.679636 17071 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
I20260928 23:50:34.708112 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:42953
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:50:34.817559 17077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:34.817852 17077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:34.817973 17077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:34.821946 17077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:34.822031 17077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:34.822114 17077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:50:34.827334 17077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:42953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:34.828346 17077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:34.829524 17077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:34.838935 17085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:34.839023 17083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:34.838932 17082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:34.839178 17077 server_base.cc:1034] running on GCE node
I20260928 23:50:34.839653 17077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:34.840272 17077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:34.841455 17077 hybrid_clock.cc:648] HybridClock initialized: now 1790639434841427 us; error 45 us; skew 500 ppm
Sep 28 23:50:34 dist-test-slave-p5bw krb5kdc[16970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639434, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:34.844170 17077 init.cc:377] Logged in from keytab as kudu/127.6.152.1@KRBTEST.COM (short username kudu)
I20260928 23:50:34.845031 17077 webserver.cc:460] Webserver started at http://127.6.152.1:34315/ using document root <none> and password file <none>
I20260928 23:50:34.845577 17077 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:34.845675 17077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:34.845944 17077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:34.847927 17077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "23f06eb7a7e44b66b6e35067de1c6ccf"
format_stamp: "Formatted at 2026-09-28 23:50:34 on dist-test-slave-p5bw"
server_key: "93638983f15b543f1cc4a3963290f442"
server_key_iv: "f21d50a7b1e27f91b24ad1752334da66"
server_key_version: "encryptionkey@0"
I20260928 23:50:34.848416 17077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "23f06eb7a7e44b66b6e35067de1c6ccf"
format_stamp: "Formatted at 2026-09-28 23:50:34 on dist-test-slave-p5bw"
server_key: "93638983f15b543f1cc4a3963290f442"
server_key_iv: "f21d50a7b1e27f91b24ad1752334da66"
server_key_version: "encryptionkey@0"
I20260928 23:50:34.851857 17077 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260928 23:50:34.854146 17092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:34.855309 17077 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:50:34.855397 17077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "23f06eb7a7e44b66b6e35067de1c6ccf"
format_stamp: "Formatted at 2026-09-28 23:50:34 on dist-test-slave-p5bw"
server_key: "93638983f15b543f1cc4a3963290f442"
server_key_iv: "f21d50a7b1e27f91b24ad1752334da66"
server_key_version: "encryptionkey@0"
I20260928 23:50:34.855476 17077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:34.884310 17077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:34.887215 17077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:34.887374 17077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:34.887909 17077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:50:34.888669 17077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:34.888747 17077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:34.888811 17077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:34.888875 17077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:34.898394 17077 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:35301
I20260928 23:50:34.898418 17206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:35301 every 8 connection(s)
I20260928 23:50:34.899319 17077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:50:34.904572 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 17077
I20260928 23:50:34.904688 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
I20260928 23:50:34.905025 6752 external_mini_cluster.cc:1428] Setting key b949a3a9db717e1536ee89bc18bade68
Sep 28 23:50:34 dist-test-slave-p5bw krb5kdc[16970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639434, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:50:34.914041 17054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:34.901117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:53899 (local address 127.6.152.62:42953)
0928 23:50:34.901254 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:34.901258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:34.902141 (+ 883us) server_negotiation.cc:407] Connection header received
0928 23:50:34.903469 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:34.903472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:34.903519 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:34.903598 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:34.904905 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:34.905544 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:34.906989 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:34.907210 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:34.907291 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:34.909961 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:34.909967 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:34.909973 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:34.910006 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:34.911677 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:34.912344 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:34.912347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:34.912348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:34.912424 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:34.912871 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:34.912875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:34.912876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:34.913062 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:34.913142 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:34.913790 (+ 648us) server_negotiation.cc:299] Negotiation successful
0928 23:50:34.913894 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.6.152.2@KRBTEST.COM; defaulting to no policy
I20260928 23:50:34.915251 17210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:34.901556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:42953 (local address 127.6.152.1:53899)
0928 23:50:34.901984 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:34.902025 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:34.903266 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:34.903750 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:34.903756 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:34.904095 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:34.904731 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:34.904753 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:34.905662 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:34.905666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:34.906870 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:34.906884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:34.907355 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:34.907361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:34.907601 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:34.908310 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:34.908325 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:34.909789 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:34.911798 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:34.911807 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:34.911825 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:34.912234 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:34.912541 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:34.912544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:34.912546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:34.912727 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:34.913173 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:34.913184 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:34.913522 (+ 338us) client_negotiation.cc:769] Sending connection context
0928 23:50:34.913748 (+ 226us) client_negotiation.cc:240] Negotiation successful
0928 23:50:34.914109 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":108,"threads_started":1}
I20260928 23:50:34.916568 17207 heartbeater.cc:347] Connected to a master server at 127.6.152.62:42953
I20260928 23:50:34.916778 17207 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:34.917320 17207 heartbeater.cc:511] Master 127.6.152.62:42953 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.2@KRBTEST.COM" created.
I20260928 23:50:34.919044 17015 ts_manager.cc:194] Registered new tserver with Master: 23f06eb7a7e44b66b6e35067de1c6ccf (127.6.152.1:35301)
I20260928 23:50:34.920380 17015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.1@KRBTEST.COM'} at 127.6.152.1:53899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
I20260928 23:50:34.935482 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:42953
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:50:35.046172 17213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:35.046418 17213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:35.046496 17213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:35.050448 17213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:35.050508 17213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:35.050611 17213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:50:35.055805 17213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:42953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:35.056762 17213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:35.057703 17213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:35.067490 17221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:35.067530 17218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:35.067510 17219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:35.067871 17213 server_base.cc:1034] running on GCE node
I20260928 23:50:35.068295 17213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:35.068979 17213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:35.070152 17213 hybrid_clock.cc:648] HybridClock initialized: now 1790639435070144 us; error 31 us; skew 500 ppm
Sep 28 23:50:35 dist-test-slave-p5bw krb5kdc[16970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639435, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:35.072762 17213 init.cc:377] Logged in from keytab as kudu/127.6.152.2@KRBTEST.COM (short username kudu)
I20260928 23:50:35.073559 17213 webserver.cc:460] Webserver started at http://127.6.152.2:37171/ using document root <none> and password file <none>
I20260928 23:50:35.074050 17213 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:35.074106 17213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:35.074266 17213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:35.076143 17213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "2e015a3da0bd442bbc696193a5dc5972"
format_stamp: "Formatted at 2026-09-28 23:50:35 on dist-test-slave-p5bw"
server_key: "195275375afe5ed457d602d160a232d6"
server_key_iv: "c0e2f491b6af5acd9066b8bad6db3058"
server_key_version: "encryptionkey@0"
I20260928 23:50:35.076624 17213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "2e015a3da0bd442bbc696193a5dc5972"
format_stamp: "Formatted at 2026-09-28 23:50:35 on dist-test-slave-p5bw"
server_key: "195275375afe5ed457d602d160a232d6"
server_key_iv: "c0e2f491b6af5acd9066b8bad6db3058"
server_key_version: "encryptionkey@0"
I20260928 23:50:35.080096 17213 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260928 23:50:35.082410 17228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:35.083567 17213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:50:35.083693 17213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "2e015a3da0bd442bbc696193a5dc5972"
format_stamp: "Formatted at 2026-09-28 23:50:35 on dist-test-slave-p5bw"
server_key: "195275375afe5ed457d602d160a232d6"
server_key_iv: "c0e2f491b6af5acd9066b8bad6db3058"
server_key_version: "encryptionkey@0"
I20260928 23:50:35.083789 17213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:35.114238 17213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:35.117070 17213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:35.117272 17213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:35.117863 17213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:50:35.118686 17213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:35.118731 17213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:35.118800 17213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:35.118841 17213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:35.127715 17213 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:46371
I20260928 23:50:35.127737 17341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:46371 every 8 connection(s)
I20260928 23:50:35.128577 17213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:50:35.132105 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 17213
I20260928 23:50:35.132238 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:50:35.132551 6752 external_mini_cluster.cc:1428] Setting key 33785f1d70d474fe7dfc28fb4a8818fc
Sep 28 23:50:35 dist-test-slave-p5bw krb5kdc[16970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639435, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.152.3@KRBTEST.COM; defaulting to no policy
I20260928 23:50:35.143311 17054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:35.130379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:35487 (local address 127.6.152.62:42953)
0928 23:50:35.130520 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:35.130523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:35.131332 (+ 809us) server_negotiation.cc:407] Connection header received
0928 23:50:35.133040 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:35.133045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:35.133118 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:35.133202 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:35.134683 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.135194 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.136426 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.136675 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.136756 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:35.139245 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:35.139250 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:35.139256 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:35.139292 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:35.141090 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:35.141569 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:35.141573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:35.141575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:35.141667 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:35.141987 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:35.141992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:35.141994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:35.142244 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:35.142347 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:35.142968 (+ 621us) server_negotiation.cc:299] Negotiation successful
0928 23:50:35.143109 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:50:35.144017 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:35.130761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:42953 (local address 127.6.152.2:35487)
0928 23:50:35.131190 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:35.131232 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:35.132759 (+ 1527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:35.133394 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:35.133400 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:35.133833 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:35.134527 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:35.134548 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.135313 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.135320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:35.136307 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:35.136316 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.136845 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.136851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:35.137003 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:35.137599 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:35.137616 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:35.139078 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:35.141194 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:35.141199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:35.141210 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:35.141449 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:35.141764 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:35.141767 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:35.141768 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:35.141893 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:35.142354 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:35.142359 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:35.142618 (+ 259us) client_negotiation.cc:769] Sending connection context
0928 23:50:35.142874 (+ 256us) client_negotiation.cc:240] Negotiation successful
0928 23:50:35.143136 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.3@KRBTEST.COM" created.
I20260928 23:50:35.145236 17342 heartbeater.cc:347] Connected to a master server at 127.6.152.62:42953
I20260928 23:50:35.145478 17342 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:35.146075 17342 heartbeater.cc:511] Master 127.6.152.62:42953 requested a full tablet report, sending...
I20260928 23:50:35.147122 17015 ts_manager.cc:194] Registered new tserver with Master: 2e015a3da0bd442bbc696193a5dc5972 (127.6.152.2:46371)
I20260928 23:50:35.147716 17015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.2@KRBTEST.COM'} at 127.6.152.2:35487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
I20260928 23:50:35.161468 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:42953
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:50:35.269619 17348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:50:35.269892 17348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:50:35.269965 17348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:50:35.273912 17348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:50:35.273988 17348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:50:35.274075 17348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:50:35.279681 17348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:42953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:50:35.280864 17348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:50:35.281981 17348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:50:35.291211 17356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:35.291237 17354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:50:35.291326 17353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:50:35.291646 17348 server_base.cc:1034] running on GCE node
I20260928 23:50:35.291996 17348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:50:35.292577 17348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:50:35.293756 17348 hybrid_clock.cc:648] HybridClock initialized: now 1790639435293738 us; error 29 us; skew 500 ppm
Sep 28 23:50:35 dist-test-slave-p5bw krb5kdc[16970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639435, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:50:35.296511 17348 init.cc:377] Logged in from keytab as kudu/127.6.152.3@KRBTEST.COM (short username kudu)
I20260928 23:50:35.297380 17348 webserver.cc:460] Webserver started at http://127.6.152.3:40869/ using document root <none> and password file <none>
I20260928 23:50:35.297865 17348 fs_manager.cc:359] Metadata directory not provided
I20260928 23:50:35.297920 17348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:50:35.298074 17348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:50:35.299942 17348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "a2c8a13b62574b4bbfb59213293bf996"
format_stamp: "Formatted at 2026-09-28 23:50:35 on dist-test-slave-p5bw"
server_key: "30cb4eec12ed47138aa8107aecc37b81"
server_key_iv: "49e65092f60a54ddf8faefdfda29b61c"
server_key_version: "encryptionkey@0"
I20260928 23:50:35.300426 17348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "a2c8a13b62574b4bbfb59213293bf996"
format_stamp: "Formatted at 2026-09-28 23:50:35 on dist-test-slave-p5bw"
server_key: "30cb4eec12ed47138aa8107aecc37b81"
server_key_iv: "49e65092f60a54ddf8faefdfda29b61c"
server_key_version: "encryptionkey@0"
I20260928 23:50:35.304029 17348 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 23:50:35.306298 17363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:35.307452 17348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:50:35.307541 17348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "a2c8a13b62574b4bbfb59213293bf996"
format_stamp: "Formatted at 2026-09-28 23:50:35 on dist-test-slave-p5bw"
server_key: "30cb4eec12ed47138aa8107aecc37b81"
server_key_iv: "49e65092f60a54ddf8faefdfda29b61c"
server_key_version: "encryptionkey@0"
I20260928 23:50:35.307622 17348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:50:35.325709 17348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:50:35.328425 17348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:50:35.328572 17348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:50:35.329116 17348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:50:35.329849 17348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:50:35.329895 17348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:35.329933 17348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:50:35.329954 17348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:50:35.338327 17348 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:37777
I20260928 23:50:35.338351 17477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:37777 every 8 connection(s)
I20260928 23:50:35.339174 17348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:50:35.347669 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 17348
I20260928 23:50:35.347810 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:50:35.348117 6752 external_mini_cluster.cc:1428] Setting key 1ae164c638c76d39a0823a50c6e951ab
Sep 28 23:50:35 dist-test-slave-p5bw krb5kdc[16970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639435, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:50:35.352348 17054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:35.340899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:58889 (local address 127.6.152.62:42953)
0928 23:50:35.341046 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:35.341051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:35.341919 (+ 868us) server_negotiation.cc:407] Connection header received
0928 23:50:35.343242 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:35.343245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:35.343292 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:35.343354 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:35.344442 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.344980 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.346045 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.346251 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.346324 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:35.349016 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:35.349020 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:35.349025 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:35.349097 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:35.350670 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:35.351117 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:35.351121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:35.351122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:35.351196 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:35.351471 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:35.351473 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:35.351474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:35.351654 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:35.351727 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:35.352139 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 23:50:35.352232 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:50:35.353246 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:35.341306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:42953 (local address 127.6.152.3:58889)
0928 23:50:35.341784 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:35.341824 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:35.343048 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:35.343511 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:35.343517 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:35.343848 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:50:35.344323 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:35.344335 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.345080 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.345083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:35.345948 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:35.345957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.346367 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.346370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:35.346510 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:35.347088 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:50:35.347103 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:50:35.348873 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:50:35.350769 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:35.350773 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:35.350784 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:35.351024 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:35.351273 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:50:35.351275 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:50:35.351277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:50:35.351397 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:50:35.351735 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:50:35.351740 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:50:35.351950 (+ 210us) client_negotiation.cc:769] Sending connection context
0928 23:50:35.352093 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 23:50:35.352335 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":126,"threads_started":1}
I20260928 23:50:35.354285 17478 heartbeater.cc:347] Connected to a master server at 127.6.152.62:42953
I20260928 23:50:35.354560 17478 heartbeater.cc:464] Registering TS with master...
I20260928 23:50:35.355209 17478 heartbeater.cc:511] Master 127.6.152.62:42953 requested a full tablet report, sending...
I20260928 23:50:35.356217 17015 ts_manager.cc:194] Registered new tserver with Master: a2c8a13b62574b4bbfb59213293bf996 (127.6.152.3:37777)
I20260928 23:50:35.356769 17015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.3@KRBTEST.COM'} at 127.6.152.3:58889
I20260928 23:50:35.361879 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:50:35.376241 17054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:35.363805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55462 (local address 127.6.152.62:42953)
0928 23:50:35.363926 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:35.363929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:35.364169 (+ 240us) server_negotiation.cc:407] Connection header received
0928 23:50:35.364314 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:35.364317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:35.364361 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:35.364451 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:50:35.365204 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.365721 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.366555 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.366791 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.366898 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:35.368110 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:50:35.368114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:50:35.368117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:50:35.368141 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:50:35.374501 (+ 6360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:35.374961 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:35.374964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:35.374965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:35.375037 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:50:35.375318 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:50:35.375321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:50:35.375323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:50:35.375568 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:50:35.375716 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:35.375941 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 23:50:35.376078 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:50:35.378968 17015 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:50:35.382086 17015 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:50:35.395829 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:35.390044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:46371 (local address 127.0.0.1:41202)
0928 23:50:35.390408 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:35.390459 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:35.390583 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:35.391374 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:35.391380 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:35.391403 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:35.391557 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:35.391568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.392933 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.392937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:35.394268 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:35.394278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.395297 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.395301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:35.395401 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:35.395441 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 23:50:35.395554 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 23:50:35.395676 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"spinlock_wait_cycles":2432,"thread_start_us":112,"threads_started":1}
I20260928 23:50:35.395831 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:35.390210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41202 (local address 127.6.152.2:46371)
0928 23:50:35.390638 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:35.390649 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:35.390695 (+ 46us) server_negotiation.cc:407] Connection header received
0928 23:50:35.390779 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:35.390785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:35.391005 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:35.391170 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:35.391690 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.392790 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.394398 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.395187 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.395300 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:35.395439 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:35.395579 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 23:50:35.395684 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":116,"threads_started":1}
I20260928 23:50:35.396744 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:35.390793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:35301 (local address 127.0.0.1:33152)
0928 23:50:35.391349 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:35.391362 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:35.391466 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:35.392466 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:35.392472 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:35.392494 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:35.392584 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:35.392592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.393658 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.393665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:35.394945 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:35.394955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.396419 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.396422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:35.396495 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:35.396512 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 23:50:35.396561 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 23:50:35.396610 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":1,"spinlock_wait_cycles":2304,"thread_start_us":92,"threads_started":1}
I20260928 23:50:35.397028 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:35.390942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33152 (local address 127.6.152.1:35301)
0928 23:50:35.391947 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:35.391962 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:35.391985 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:50:35.392055 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:35.392060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:35.392246 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:35.392389 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:35.392705 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.393525 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.395072 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.396312 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.396415 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:35.396538 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:35.396586 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 23:50:35.396668 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":902,"thread_start_us":108,"threads_started":1}
I20260928 23:50:35.397198 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:35.391128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:37777 (local address 127.0.0.1:56708)
0928 23:50:35.391511 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:35.391525 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:35.391602 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:35.392293 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:35.392296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:35.392313 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:35.392421 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:35.392431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.393860 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.393863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:35.395028 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:35.395037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.396762 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.396765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:35.396893 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:35.396914 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 23:50:35.396986 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 23:50:35.397059 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":92,"threads_started":1}
I20260928 23:50:35.397275 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:35.391300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56708 (local address 127.6.152.3:37777)
0928 23:50:35.391889 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:35.391902 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:35.391925 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:50:35.392002 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:35.392007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:35.392163 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:35.392303 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:35.392549 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.393734 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.395126 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.395965 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.396065 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:35.396212 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:35.397036 (+ 824us) server_negotiation.cc:299] Negotiation successful
0928 23:50:35.397145 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":114,"threads_started":1}
I20260928 23:50:35.398768 17276 tablet_service.cc:1467] Processing CreateTablet for tablet 5b98faf1bb50418f8161e4b6e85e576e (DEFAULT_TABLE table=test-table [id=855c5ce5a6b34cecab23b79551757698]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:50:35.399745 17276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b98faf1bb50418f8161e4b6e85e576e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:35.401850 17141 tablet_service.cc:1467] Processing CreateTablet for tablet 5b98faf1bb50418f8161e4b6e85e576e (DEFAULT_TABLE table=test-table [id=855c5ce5a6b34cecab23b79551757698]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:50:35.402101 17411 tablet_service.cc:1467] Processing CreateTablet for tablet 5b98faf1bb50418f8161e4b6e85e576e (DEFAULT_TABLE table=test-table [id=855c5ce5a6b34cecab23b79551757698]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:50:35.402806 17141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b98faf1bb50418f8161e4b6e85e576e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:35.403084 17411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b98faf1bb50418f8161e4b6e85e576e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:50:35.407130 17495 tablet_bootstrap.cc:492] T 5b98faf1bb50418f8161e4b6e85e576e P 2e015a3da0bd442bbc696193a5dc5972: Bootstrap starting.
I20260928 23:50:35.409627 17497 tablet_bootstrap.cc:492] T 5b98faf1bb50418f8161e4b6e85e576e P a2c8a13b62574b4bbfb59213293bf996: Bootstrap starting.
I20260928 23:50:35.410264 17495 tablet_bootstrap.cc:654] T 5b98faf1bb50418f8161e4b6e85e576e P 2e015a3da0bd442bbc696193a5dc5972: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:35.411034 17495 log.cc:824] T 5b98faf1bb50418f8161e4b6e85e576e P 2e015a3da0bd442bbc696193a5dc5972: Log is configured to *not* fsync() on all Append() calls
I20260928 23:50:35.411736 17496 tablet_bootstrap.cc:492] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf: Bootstrap starting.
I20260928 23:50:35.412073 17497 tablet_bootstrap.cc:654] T 5b98faf1bb50418f8161e4b6e85e576e P a2c8a13b62574b4bbfb59213293bf996: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:35.412722 17497 log.cc:824] T 5b98faf1bb50418f8161e4b6e85e576e P a2c8a13b62574b4bbfb59213293bf996: Log is configured to *not* fsync() on all Append() calls
I20260928 23:50:35.413000 17495 tablet_bootstrap.cc:492] T 5b98faf1bb50418f8161e4b6e85e576e P 2e015a3da0bd442bbc696193a5dc5972: No bootstrap required, opened a new log
I20260928 23:50:35.413197 17495 ts_tablet_manager.cc:1397] T 5b98faf1bb50418f8161e4b6e85e576e P 2e015a3da0bd442bbc696193a5dc5972: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:50:35.414525 17497 tablet_bootstrap.cc:492] T 5b98faf1bb50418f8161e4b6e85e576e P a2c8a13b62574b4bbfb59213293bf996: No bootstrap required, opened a new log
I20260928 23:50:35.414703 17497 ts_tablet_manager.cc:1397] T 5b98faf1bb50418f8161e4b6e85e576e P a2c8a13b62574b4bbfb59213293bf996: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 23:50:35.416494 17495 raft_consensus.cc:348] T 5b98faf1bb50418f8161e4b6e85e576e P 2e015a3da0bd442bbc696193a5dc5972 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e015a3da0bd442bbc696193a5dc5972" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46371 } } peers { permanent_uuid: "23f06eb7a7e44b66b6e35067de1c6ccf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 35301 } } peers { permanent_uuid: "a2c8a13b62574b4bbfb59213293bf996" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 37777 } }
I20260928 23:50:35.416740 17495 raft_consensus.cc:374] T 5b98faf1bb50418f8161e4b6e85e576e P 2e015a3da0bd442bbc696193a5dc5972 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:35.416798 17495 raft_consensus.cc:729] T 5b98faf1bb50418f8161e4b6e85e576e P 2e015a3da0bd442bbc696193a5dc5972 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e015a3da0bd442bbc696193a5dc5972, State: Initialized, Role: FOLLOWER
I20260928 23:50:35.417258 17495 consensus_queue.cc:260] T 5b98faf1bb50418f8161e4b6e85e576e P 2e015a3da0bd442bbc696193a5dc5972 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e015a3da0bd442bbc696193a5dc5972" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46371 } } peers { permanent_uuid: "23f06eb7a7e44b66b6e35067de1c6ccf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 35301 } } peers { permanent_uuid: "a2c8a13b62574b4bbfb59213293bf996" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 37777 } }
I20260928 23:50:35.417560 17497 raft_consensus.cc:348] T 5b98faf1bb50418f8161e4b6e85e576e P a2c8a13b62574b4bbfb59213293bf996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e015a3da0bd442bbc696193a5dc5972" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46371 } } peers { permanent_uuid: "23f06eb7a7e44b66b6e35067de1c6ccf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 35301 } } peers { permanent_uuid: "a2c8a13b62574b4bbfb59213293bf996" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 37777 } }
I20260928 23:50:35.417883 17342 heartbeater.cc:503] Master 127.6.152.62:42953 was elected leader, sending a full tablet report...
I20260928 23:50:35.418226 17495 ts_tablet_manager.cc:1428] T 5b98faf1bb50418f8161e4b6e85e576e P 2e015a3da0bd442bbc696193a5dc5972: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260928 23:50:35.418780 17497 raft_consensus.cc:374] T 5b98faf1bb50418f8161e4b6e85e576e P a2c8a13b62574b4bbfb59213293bf996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:35.418867 17497 raft_consensus.cc:729] T 5b98faf1bb50418f8161e4b6e85e576e P a2c8a13b62574b4bbfb59213293bf996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c8a13b62574b4bbfb59213293bf996, State: Initialized, Role: FOLLOWER
I20260928 23:50:35.419270 17497 consensus_queue.cc:260] T 5b98faf1bb50418f8161e4b6e85e576e P a2c8a13b62574b4bbfb59213293bf996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e015a3da0bd442bbc696193a5dc5972" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46371 } } peers { permanent_uuid: "23f06eb7a7e44b66b6e35067de1c6ccf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 35301 } } peers { permanent_uuid: "a2c8a13b62574b4bbfb59213293bf996" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 37777 } }
I20260928 23:50:35.419853 17478 heartbeater.cc:503] Master 127.6.152.62:42953 was elected leader, sending a full tablet report...
I20260928 23:50:35.420478 17496 tablet_bootstrap.cc:654] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf: Neither blocks nor log segments found. Creating new log.
I20260928 23:50:35.421033 17497 ts_tablet_manager.cc:1428] T 5b98faf1bb50418f8161e4b6e85e576e P a2c8a13b62574b4bbfb59213293bf996: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:50:35.421383 17496 log.cc:824] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf: Log is configured to *not* fsync() on all Append() calls
I20260928 23:50:35.423475 17496 tablet_bootstrap.cc:492] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf: No bootstrap required, opened a new log
I20260928 23:50:35.423671 17496 ts_tablet_manager.cc:1397] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260928 23:50:35.426822 17496 raft_consensus.cc:348] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e015a3da0bd442bbc696193a5dc5972" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46371 } } peers { permanent_uuid: "23f06eb7a7e44b66b6e35067de1c6ccf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 35301 } } peers { permanent_uuid: "a2c8a13b62574b4bbfb59213293bf996" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 37777 } }
I20260928 23:50:35.427028 17496 raft_consensus.cc:374] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:50:35.427062 17496 raft_consensus.cc:729] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23f06eb7a7e44b66b6e35067de1c6ccf, State: Initialized, Role: FOLLOWER
I20260928 23:50:35.427400 17496 consensus_queue.cc:260] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e015a3da0bd442bbc696193a5dc5972" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46371 } } peers { permanent_uuid: "23f06eb7a7e44b66b6e35067de1c6ccf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 35301 } } peers { permanent_uuid: "a2c8a13b62574b4bbfb59213293bf996" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 37777 } }
I20260928 23:50:35.427974 17207 heartbeater.cc:503] Master 127.6.152.62:42953 was elected leader, sending a full tablet report...
I20260928 23:50:35.428277 17496 ts_tablet_manager.cc:1428] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 23:50:35.451576 17503 raft_consensus.cc:482] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:50:35.451776 17503 raft_consensus.cc:504] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e015a3da0bd442bbc696193a5dc5972" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46371 } } peers { permanent_uuid: "23f06eb7a7e44b66b6e35067de1c6ccf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 35301 } } peers { permanent_uuid: "a2c8a13b62574b4bbfb59213293bf996" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 37777 } }
I20260928 23:50:35.453038 17503 leader_election.cc:288] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e015a3da0bd442bbc696193a5dc5972 (127.6.152.2:46371), a2c8a13b62574b4bbfb59213293bf996 (127.6.152.3:37777)
I20260928 23:50:35.458118 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:35.453853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:57689 (local address 127.6.152.3:37777)
0928 23:50:35.453985 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:35.453992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:35.454075 (+ 83us) server_negotiation.cc:407] Connection header received
0928 23:50:35.454206 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:35.454208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:35.454248 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:35.454306 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:35.454567 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.455279 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.456851 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.457458 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.457566 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:35.457641 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:35.457921 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 23:50:35.457989 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:50:35.458125 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:35.453412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:55321 (local address 127.6.152.2:46371)
0928 23:50:35.453544 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:35.453548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:35.453724 (+ 176us) server_negotiation.cc:407] Connection header received
0928 23:50:35.453875 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:35.453880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:35.453937 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:35.454023 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:50:35.454378 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.455175 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.456850 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.457458 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.457566 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:35.457628 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:35.457921 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 23:50:35.457991 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:50:35.458181 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:35.453706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:37777 (local address 127.6.152.1:57689)
0928 23:50:35.453974 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:35.454003 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:35.454101 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:35.454343 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:35.454349 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:35.454369 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:35.454468 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:35.454472 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.455380 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.455382 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:35.456713 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:35.456723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.457588 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.457592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:35.457769 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:35.457817 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 23:50:35.457891 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 23:50:35.457994 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":137,"spinlock_wait_cycles":2304,"thread_start_us":91,"threads_started":1}
I20260928 23:50:35.458181 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:35.453314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:46371 (local address 127.6.152.1:55321)
0928 23:50:35.453630 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:35.453646 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:50:35.453752 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:50:35.454052 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:50:35.454057 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:50:35.454097 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:50:35.454247 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:35.454253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.455277 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.455279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:35.456716 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:50:35.456723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.457588 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.457592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:50:35.457773 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:35.457814 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 23:50:35.457891 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 23:50:35.457994 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":172,"spinlock_wait_cycles":28928,"thread_start_us":120,"threads_started":1}
I20260928 23:50:35.458813 17296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b98faf1bb50418f8161e4b6e85e576e" candidate_uuid: "23f06eb7a7e44b66b6e35067de1c6ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e015a3da0bd442bbc696193a5dc5972" is_pre_election: true
I20260928 23:50:35.458819 17431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b98faf1bb50418f8161e4b6e85e576e" candidate_uuid: "23f06eb7a7e44b66b6e35067de1c6ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c8a13b62574b4bbfb59213293bf996" is_pre_election: true
I20260928 23:50:35.459141 17431 raft_consensus.cc:2454] T 5b98faf1bb50418f8161e4b6e85e576e P a2c8a13b62574b4bbfb59213293bf996 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23f06eb7a7e44b66b6e35067de1c6ccf in term 0.
I20260928 23:50:35.459141 17296 raft_consensus.cc:2454] T 5b98faf1bb50418f8161e4b6e85e576e P 2e015a3da0bd442bbc696193a5dc5972 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23f06eb7a7e44b66b6e35067de1c6ccf in term 0.
I20260928 23:50:35.459589 17096 leader_election.cc:302] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23f06eb7a7e44b66b6e35067de1c6ccf, 2e015a3da0bd442bbc696193a5dc5972; no voters:
I20260928 23:50:35.459812 17503 raft_consensus.cc:2781] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:50:35.459884 17503 raft_consensus.cc:482] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:50:35.459915 17503 raft_consensus.cc:3037] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:35.461001 17503 raft_consensus.cc:504] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e015a3da0bd442bbc696193a5dc5972" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46371 } } peers { permanent_uuid: "23f06eb7a7e44b66b6e35067de1c6ccf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 35301 } } peers { permanent_uuid: "a2c8a13b62574b4bbfb59213293bf996" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 37777 } }
I20260928 23:50:35.461318 17503 leader_election.cc:288] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf [CANDIDATE]: Term 1 election: Requested vote from peers 2e015a3da0bd442bbc696193a5dc5972 (127.6.152.2:46371), a2c8a13b62574b4bbfb59213293bf996 (127.6.152.3:37777)
I20260928 23:50:35.461654 17296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b98faf1bb50418f8161e4b6e85e576e" candidate_uuid: "23f06eb7a7e44b66b6e35067de1c6ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e015a3da0bd442bbc696193a5dc5972"
I20260928 23:50:35.461699 17431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b98faf1bb50418f8161e4b6e85e576e" candidate_uuid: "23f06eb7a7e44b66b6e35067de1c6ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c8a13b62574b4bbfb59213293bf996"
I20260928 23:50:35.461748 17296 raft_consensus.cc:3037] T 5b98faf1bb50418f8161e4b6e85e576e P 2e015a3da0bd442bbc696193a5dc5972 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:35.461840 17431 raft_consensus.cc:3037] T 5b98faf1bb50418f8161e4b6e85e576e P a2c8a13b62574b4bbfb59213293bf996 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:50:35.462809 17296 raft_consensus.cc:2454] T 5b98faf1bb50418f8161e4b6e85e576e P 2e015a3da0bd442bbc696193a5dc5972 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23f06eb7a7e44b66b6e35067de1c6ccf in term 1.
I20260928 23:50:35.462986 17431 raft_consensus.cc:2454] T 5b98faf1bb50418f8161e4b6e85e576e P a2c8a13b62574b4bbfb59213293bf996 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23f06eb7a7e44b66b6e35067de1c6ccf in term 1.
I20260928 23:50:35.463094 17096 leader_election.cc:302] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23f06eb7a7e44b66b6e35067de1c6ccf, 2e015a3da0bd442bbc696193a5dc5972; no voters:
I20260928 23:50:35.463305 17503 raft_consensus.cc:2781] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:50:35.463550 17503 raft_consensus.cc:686] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf [term 1 LEADER]: Becoming Leader. State: Replica: 23f06eb7a7e44b66b6e35067de1c6ccf, State: Running, Role: LEADER
I20260928 23:50:35.463889 17503 consensus_queue.cc:237] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e015a3da0bd442bbc696193a5dc5972" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46371 } } peers { permanent_uuid: "23f06eb7a7e44b66b6e35067de1c6ccf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 35301 } } peers { permanent_uuid: "a2c8a13b62574b4bbfb59213293bf996" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 37777 } }
I20260928 23:50:35.466825 17014 catalog_manager.cc:5495] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf reported cstate change: term changed from 0 to 1, leader changed from <none> to 23f06eb7a7e44b66b6e35067de1c6ccf (127.6.152.1). New cstate: current_term: 1 leader_uuid: "23f06eb7a7e44b66b6e35067de1c6ccf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e015a3da0bd442bbc696193a5dc5972" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23f06eb7a7e44b66b6e35067de1c6ccf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 35301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2c8a13b62574b4bbfb59213293bf996" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 37777 } health_report { overall_health: UNKNOWN } } }
I20260928 23:50:35.495729 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:35.492451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33160 (local address 127.6.152.1:35301)
0928 23:50:35.492574 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 23:50:35.492577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:50:35.492590 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:50:35.492678 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:50:35.492680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:50:35.492735 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:50:35.492866 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:50:35.493072 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.493556 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.494377 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:50:35.494595 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:50:35.494674 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:50:35.494921 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:50:35.495364 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:50:35.495493 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 23:50:35.495572 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 23:50:35.495623 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:50:35.501282 17431 raft_consensus.cc:1261] T 5b98faf1bb50418f8161e4b6e85e576e P a2c8a13b62574b4bbfb59213293bf996 [term 1 FOLLOWER]: Refusing update from remote peer 23f06eb7a7e44b66b6e35067de1c6ccf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:50:35.501307 17296 raft_consensus.cc:1261] T 5b98faf1bb50418f8161e4b6e85e576e P 2e015a3da0bd442bbc696193a5dc5972 [term 1 FOLLOWER]: Refusing update from remote peer 23f06eb7a7e44b66b6e35067de1c6ccf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:50:35.501971 17503 consensus_queue.cc:1035] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e015a3da0bd442bbc696193a5dc5972" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 46371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:50:35.502146 17506 consensus_queue.cc:1035] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c8a13b62574b4bbfb59213293bf996" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 37777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:50:35.516436 17015 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:50:35.516599 17015 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:50:35.518863 17015 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad2feae16b184a7db3fbfccc28e3b2b3: Sending DeleteTablet for 3 replicas of tablet 5b98faf1bb50418f8161e4b6e85e576e
I20260928 23:50:35.519678 17276 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b98faf1bb50418f8161e4b6e85e576e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:50:35 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:41202
I20260928 23:50:35.519699 17411 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b98faf1bb50418f8161e4b6e85e576e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:50:35 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:56708
I20260928 23:50:35.520179 17520 tablet_replica.cc:330] T 5b98faf1bb50418f8161e4b6e85e576e P 2e015a3da0bd442bbc696193a5dc5972: stopping tablet replica
I20260928 23:50:35.520315 17141 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b98faf1bb50418f8161e4b6e85e576e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:50:35 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:33152
I20260928 23:50:35.520609 17520 raft_consensus.cc:2229] T 5b98faf1bb50418f8161e4b6e85e576e P 2e015a3da0bd442bbc696193a5dc5972 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:50:35.520787 17521 tablet_replica.cc:330] T 5b98faf1bb50418f8161e4b6e85e576e P 23f06eb7a7e44b66b6e35067de1c6ccf: stopping tablet replica
I20260928 23:50:35.520902 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 17077
I20260928 23:50:35.521067 17520 raft_consensus.cc:2258] T 5b98faf1bb50418f8161e4b6e85e576e P 2e015a3da0bd442bbc696193a5dc5972 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:50:35.521478 17519 tablet_replica.cc:330] T 5b98faf1bb50418f8161e4b6e85e576e P a2c8a13b62574b4bbfb59213293bf996: stopping tablet replica
I20260928 23:50:35.521785 17519 raft_consensus.cc:2229] T 5b98faf1bb50418f8161e4b6e85e576e P a2c8a13b62574b4bbfb59213293bf996 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:50:35.522157 17519 raft_consensus.cc:2258] T 5b98faf1bb50418f8161e4b6e85e576e P a2c8a13b62574b4bbfb59213293bf996 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:50:35.523636 17520 ts_tablet_manager.cc:1905] T 5b98faf1bb50418f8161e4b6e85e576e P 2e015a3da0bd442bbc696193a5dc5972: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:50:35.523636 17519 ts_tablet_manager.cc:1905] T 5b98faf1bb50418f8161e4b6e85e576e P a2c8a13b62574b4bbfb59213293bf996: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:50:35.526276 17520 ts_tablet_manager.cc:1918] T 5b98faf1bb50418f8161e4b6e85e576e P 2e015a3da0bd442bbc696193a5dc5972: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:50:35.526270 17519 ts_tablet_manager.cc:1918] T 5b98faf1bb50418f8161e4b6e85e576e P a2c8a13b62574b4bbfb59213293bf996: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:50:35.526369 17519 log.cc:1196] T 5b98faf1bb50418f8161e4b6e85e576e P a2c8a13b62574b4bbfb59213293bf996: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/5b98faf1bb50418f8161e4b6e85e576e
I20260928 23:50:35.526369 17520 log.cc:1196] T 5b98faf1bb50418f8161e4b6e85e576e P 2e015a3da0bd442bbc696193a5dc5972: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639374546653-6752-0/minicluster-data/ts-1/wal/wals/5b98faf1bb50418f8161e4b6e85e576e
I20260928 23:50:35.526708 17520 ts_tablet_manager.cc:1939] T 5b98faf1bb50418f8161e4b6e85e576e P 2e015a3da0bd442bbc696193a5dc5972: Deleting consensus metadata
I20260928 23:50:35.526708 17519 ts_tablet_manager.cc:1939] T 5b98faf1bb50418f8161e4b6e85e576e P a2c8a13b62574b4bbfb59213293bf996: Deleting consensus metadata
W20260928 23:50:35.527571 17002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.152.1:35301 (error 108)
I20260928 23:50:35.528053 17002 catalog_manager.cc:4841] TS a2c8a13b62574b4bbfb59213293bf996 (127.6.152.3:37777): tablet 5b98faf1bb50418f8161e4b6e85e576e (table test-table [id=855c5ce5a6b34cecab23b79551757698]) successfully deleted
I20260928 23:50:35.528227 17002 catalog_manager.cc:4841] TS 2e015a3da0bd442bbc696193a5dc5972 (127.6.152.2:46371): tablet 5b98faf1bb50418f8161e4b6e85e576e (table test-table [id=855c5ce5a6b34cecab23b79551757698]) successfully deleted
I20260928 23:50:35.528534 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 17213
I20260928 23:50:35.528847 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:50:35.528491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:35301 (local address 127.0.0.1:33172)
0928 23:50:35.528661 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 23:50:35.528735 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.1:35301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 23:50:35.529284 17002 catalog_manager.cc:4584] TS 23f06eb7a7e44b66b6e35067de1c6ccf (127.6.152.1:35301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5b98faf1bb50418f8161e4b6e85e576e: Network error: Client connection negotiation failed: client connection to 127.6.152.1:35301: connect: Connection refused (error 111)
I20260928 23:50:35.533990 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 17348
I20260928 23:50:35.539187 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 16983
2026-09-28T23:50:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2258 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:50:35 dist-test-slave-p5bw krb5kdc[17525](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:50:35 dist-test-slave-p5bw krb5kdc[17525](info): set up 2 sockets
Sep 28 23:50:35 dist-test-slave-p5bw krb5kdc[17525](info): commencing operation
krb5kdc: starting...
W20260928 23:50:36.578521 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:50:36 dist-test-slave-p5bw krb5kdc[17525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:50:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:50:36Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.152.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.152.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.152.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/rangeradmin_127.6.152.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.152.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/rangeradmin_127.6.152.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.152.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.152.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.152.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/rangerlookup_127.6.152.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.152.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/rangerlookup_127.6.152.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.152.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.152.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.152.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/HTTP_127.6.152.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.152.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/HTTP_127.6.152.20@KRBTEST.COM.keytab.
I20260928 23:50:36.752607 6752 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskXX9CKr/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/postgres -l logfile start
2026-09-28 23:50:39.285 UTC [17555] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 23:50:39.285 UTC [17555] LOG: listening on IPv4 address "127.6.152.20", port 60579
2026-09-28 23:50:39.289 UTC [17555] LOG: listening on Unix socket "/tmp/.s.PGSQL.60579"
2026-09-28 23:50:39.324 UTC [17558] LOG: database system was shut down at 2026-09-28 23:50:37 UTC
2026-09-28 23:50:39.328 UTC [17555] LOG: database system is ready to accept connections
I20260928 23:50:40.274418 6752 mini_postgres.cc:96] Postgres bound to 60579
2026-09-28 23:50:40.281 UTC [17566] FATAL: database "slave" does not exist
127.6.152.20:60579 - accepting connections
I20260928 23:50:40.281633 6752 mini_ranger.cc:161] Starting Ranger...
I20260928 23:50:40.296219 6752 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 23:50:40.793457 6752 mini_ranger.cc:87] Created ranger Postgres database
I20260928 23:50:40.793552 6752 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ranger-admin
2026-09-28 23:50:41,060 [I] DB FLAVOR :POSTGRES
2026-09-28 23:50:41,060 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 23:50:41,060 [I] Checking connection..
2026-09-28 23:50:41,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:60579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 23:50:41,322 [I] Checking connection passed.
2026-09-28 23:50:41,322 [I] --------- Verifying version history table ---------
2026-09-28 23:50:41,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:60579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:50:41,571 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 23:50:41,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:60579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:50:41,825 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 23:50:41,826 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 23:50:41,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:60579/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 23:50:42.073 UTC [17630] WARNING: there is no transaction in progress
2026-09-28 23:50:42,079 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 23:50:42,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:60579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:50:42,332 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 23:50:42,333 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 23:50:42,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:60579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 23:50:42,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:60579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 23:50:42,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:60579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-p5bw.c.gcp-upstream.internal','N') ;"
2026-09-28 23:50:43,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:60579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 23:50:43,324 [I] Table x_portal_user does not exist in database ranger
2026-09-28 23:50:43,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:60579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 23:50:43,568 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 23:50:43,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:60579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 23:50:43,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:60579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 23:50:44,059 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 23:50:44,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:60579/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 23:50:45.017 UTC [17739] WARNING: there is no transaction in progress
2026-09-28 23:50:45.033 UTC [17739] WARNING: there is no transaction in progress
2026-09-28 23:50:45.049 UTC [17739] WARNING: there is no transaction in progress
2026-09-28 23:50:45.064 UTC [17739] WARNING: there is no transaction in progress
2026-09-28 23:50:45.080 UTC [17739] WARNING: there is no transaction in progress
2026-09-28 23:50:45.097 UTC [17739] WARNING: there is no transaction in progress
2026-09-28 23:50:45.161 UTC [17739] WARNING: there is no transaction in progress
2026-09-28 23:50:45.167 UTC [17739] WARNING: there is no transaction in progress
2026-09-28 23:50:45.183 UTC [17739] WARNING: there is no transaction in progress
2026-09-28 23:50:45.193 UTC [17739] WARNING: there is no transaction in progress
2026-09-28 23:50:45.204 UTC [17739] WARNING: there is no transaction in progress
2026-09-28 23:50:45.217 UTC [17739] WARNING: there is no transaction in progress
2026-09-28 23:50:45.226 UTC [17739] WARNING: there is no transaction in progress
2026-09-28 23:50:45.232 UTC [17739] WARNING: there is no transaction in progress
2026-09-28 23:50:45.239 UTC [17739] WARNING: there is no transaction in progress
2026-09-28 23:50:45.246 UTC [17739] WARNING: there is no transaction in progress
2026-09-28 23:50:45,900 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 23:50:45,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:60579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 23:50:46,127 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 23:50:46,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:60579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 23:50:46,384 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 23:50:46,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:60579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 23:50:46,634 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 23:50:46,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:60579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 23:50:46,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:60579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 23:50:47,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:60579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-p5bw.c.gcp-upstream.internal';"
2026-09-28 23:50:47,370 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 23:50:47,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:60579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 23:50:47,617 [I] DB_PATCHES have already been applied
I20260928 23:50:47.623335 6752 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ranger-admin:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskXX9CKr/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 23:50:47.623404 6752 mini_ranger.cc:193] Using host: 127.6.152.20
I20260928 23:50:47.626953 6752 mini_ranger.cc:227] Ranger admin URL: http://127.6.152.20:50007
Sep 28, 2026 11:50:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 11:50:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 11:50:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 11:50:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 11:50:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 11:50:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 11:50:48 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 23:50:48.728467 6752 mini_ranger.cc:160] Time spent starting Ranger: real 8.447s user 0.000s sys 0.010s
Sep 28, 2026 11:51:07 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 11:51:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 11:51:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 11:51:07 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 11:51:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 11:51:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 11:51:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 11:51:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 11:51:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 11:51:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 11:51:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 11:51:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 11:51:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 11:51:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 11:51:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 11:51:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 11:51:07 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 11:51:08 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 23:51:09.466274 6752 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.152.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.152.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.152.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/rangerkms_127.6.152.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.152.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/rangerkms_127.6.152.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.152.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.152.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.152.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/HTTP_127.6.152.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.152.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/HTTP_127.6.152.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 23:51:09 dist-test-slave-p5bw krb5kdc[17525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639469, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 23:51:09.556983 6752 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 23:51:09.567710 6752 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 23:51:10.459443 6752 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 23:51:10.459543 6752 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ranger-kms
I20260928 23:51:10.459648 6752 mini_ranger_kms.cc:227] Using postgres at 127.6.152.20:60579
2026-09-28 23:51:10,643 [I] DB FLAVOR :POSTGRES
2026-09-28 23:51:10,643 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 23:51:10,643 [I] Checking connection
2026-09-28 23:51:10,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:60579/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 23:51:10,892 [I] connection success
2026-09-28 23:51:10,892 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 23:51:10,892 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 23:51:10,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:60579/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 23:51:11,119 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 23:51:11,119 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 23:51:11,119 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 23:51:11,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:60579/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 23:51:11,404 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 23:51:11.909547 6752 mini_ranger_kms.cc:310] Created kms service
I20260928 23:51:12.116236 6752 mini_ranger_kms.cc:326] Created kudu user
I20260928 23:51:12.160732 6752 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 23:51:12 dist-test-slave-p5bw krb5kdc[17525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639469, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.152.20@KRBTEST.COM
I20260928 23:51:12.534710 6752 mini_ranger_kms.cc:384] Added ranger policy
I20260928 23:51:12.534876 6752 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ranger-kms:/tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 23:51:12.534909 6752 mini_ranger_kms.cc:240] Using host: 127.6.152.20
I20260928 23:51:12.539193 6752 mini_ranger_kms.cc:276] Ranger KMS PID: 17921
I20260928 23:51:12.539324 6752 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.152.20:57489
Sep 28, 2026 11:51:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 11:51:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 11:51:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 11:51:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 11:51:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 11:51:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 23:51:14.874441 6752 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.317s user 0.024s sys 0.144s
I20260928 23:51:14.874579 6752 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 23:51:14.874722 6752 mini_ranger_kms.cc:401] 127.6.152.20:57489/kms/v1/keys
Sep 28 23:51:14 dist-test-slave-p5bw krb5kdc[17525](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790639474, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.152.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 23:51:15 dist-test-slave-p5bw krb5kdc[17525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639474, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.152.20@KRBTEST.COM for HTTP/127.6.152.20@KRBTEST.COM
Sep 28 23:51:15 dist-test-slave-p5bw krb5kdc[17525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639474, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.152.20@KRBTEST.COM for HTTP/127.6.152.20@KRBTEST.COM
Sep 28 23:51:17 dist-test-slave-p5bw krb5kdc[17525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:51:17.988850 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:43387
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:43387
--ranger_config_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.152.20:57489/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:51:18.102108 17972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:51:18.102362 17972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:51:18.102401 17972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:51:18.106475 17972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:51:18.106530 17972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:51:18.106554 17972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:51:18.106611 17972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:51:18.106642 17972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:51:18.111989 17972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.152.20:57489/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:43387
--ranger_config_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:43387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:51:18.113058 17972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:51:18.114085 17972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:51:18.122915 17980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:51:18.123040 17978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:51:18.123114 17977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:51:18.123001 17972 server_base.cc:1034] running on GCE node
I20260928 23:51:18.123674 17972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:51:18.124781 17972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:51:18.125988 17972 hybrid_clock.cc:648] HybridClock initialized: now 1790639478125966 us; error 45 us; skew 500 ppm
Sep 28 23:51:18 dist-test-slave-p5bw krb5kdc[17525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639478, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:51:18.128650 17972 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:51:18.129549 17972 webserver.cc:460] Webserver started at http://127.6.152.62:36927/ using document root <none> and password file <none>
I20260928 23:51:18.130031 17972 fs_manager.cc:359] Metadata directory not provided
I20260928 23:51:18.130092 17972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:51:18.130254 17972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:51:18 dist-test-slave-p5bw krb5kdc[17525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639478, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for HTTP/127.6.152.20@KRBTEST.COM
I20260928 23:51:18.183991 17972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "5d4d4e31823e410f95781c3cb97da809"
format_stamp: "Formatted at 2026-09-28 23:51:18 on dist-test-slave-p5bw"
server_key: "899b92a4e36434876dfe90745d346b81"
server_key_iv: "1e80151bd51b59fd995909eb5342b06e"
server_key_version: "kuduclusterkey@0"
I20260928 23:51:18.184705 17972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "5d4d4e31823e410f95781c3cb97da809"
format_stamp: "Formatted at 2026-09-28 23:51:18 on dist-test-slave-p5bw"
server_key: "899b92a4e36434876dfe90745d346b81"
server_key_iv: "1e80151bd51b59fd995909eb5342b06e"
server_key_version: "kuduclusterkey@0"
I20260928 23:51:18.189685 17972 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 23:51:18.207360 17987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:51:18.209062 17972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:51:18.209197 17972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "5d4d4e31823e410f95781c3cb97da809"
format_stamp: "Formatted at 2026-09-28 23:51:18 on dist-test-slave-p5bw"
server_key: "899b92a4e36434876dfe90745d346b81"
server_key_iv: "1e80151bd51b59fd995909eb5342b06e"
server_key_version: "kuduclusterkey@0"
I20260928 23:51:18.209333 17972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:51:18.239356 17972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:51:18.242489 17972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:51:18.242661 17972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:51:18.250016 17972 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:43387
I20260928 23:51:18.250038 18039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:43387 every 8 connection(s)
I20260928 23:51:18.251019 17972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:51:18.254285 18040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:51:18.256973 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 17972
I20260928 23:51:18.257102 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:51:18.257432 6752 external_mini_cluster.cc:1428] Setting key a3b1b88ec94e1ead47d4ba5e771e41ab
I20260928 23:51:18.260556 18040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d4d4e31823e410f95781c3cb97da809: Bootstrap starting.
I20260928 23:51:18.262964 18040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d4d4e31823e410f95781c3cb97da809: Neither blocks nor log segments found. Creating new log.
I20260928 23:51:18.263821 18040 log.cc:824] T 00000000000000000000000000000000 P 5d4d4e31823e410f95781c3cb97da809: Log is configured to *not* fsync() on all Append() calls
I20260928 23:51:18.265954 18040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d4d4e31823e410f95781c3cb97da809: No bootstrap required, opened a new log
Sep 28 23:51:18 dist-test-slave-p5bw krb5kdc[17525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:51:18.270012 18040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d4d4e31823e410f95781c3cb97da809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4d4e31823e410f95781c3cb97da809" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 43387 } }
I20260928 23:51:18.270246 18040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d4d4e31823e410f95781c3cb97da809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:51:18.270330 18040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d4d4e31823e410f95781c3cb97da809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4d4e31823e410f95781c3cb97da809, State: Initialized, Role: FOLLOWER
I20260928 23:51:18.270819 18040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d4d4e31823e410f95781c3cb97da809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4d4e31823e410f95781c3cb97da809" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 43387 } }
I20260928 23:51:18.271000 18040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d4d4e31823e410f95781c3cb97da809 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:51:18.271124 18040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d4d4e31823e410f95781c3cb97da809 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:51:18.271238 18040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d4d4e31823e410f95781c3cb97da809 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:51:18.272308 18040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d4d4e31823e410f95781c3cb97da809 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4d4e31823e410f95781c3cb97da809" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 43387 } }
I20260928 23:51:18.272629 18044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:51:18.258983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48524 (local address 127.6.152.62:43387)
0928 23:51:18.259435 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0928 23:51:18.259451 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:51:18.259488 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:51:18.260187 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:51:18.260205 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:51:18.260549 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:51:18.261084 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:51:18.261915 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:51:18.263004 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:51:18.263984 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:51:18.264510 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:51:18.264697 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:51:18.267602 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:51:18.267608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:51:18.267623 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:51:18.267650 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:51:18.269417 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:51:18.269882 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:51:18.269895 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:51:18.269900 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:51:18.270002 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:51:18.270329 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:51:18.270332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:51:18.270335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:51:18.270843 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:51:18.271068 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 23:51:18.271373 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 23:51:18.271657 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":138,"threads_started":1}
I20260928 23:51:18.272665 18040 leader_election.cc:302] T 00000000000000000000000000000000 P 5d4d4e31823e410f95781c3cb97da809 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d4d4e31823e410f95781c3cb97da809; no voters:
I20260928 23:51:18.272991 18040 leader_election.cc:288] T 00000000000000000000000000000000 P 5d4d4e31823e410f95781c3cb97da809 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:51:18.273120 18046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d4d4e31823e410f95781c3cb97da809 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:51:18.273368 18046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d4d4e31823e410f95781c3cb97da809 [term 1 LEADER]: Becoming Leader. State: Replica: 5d4d4e31823e410f95781c3cb97da809, State: Running, Role: LEADER
I20260928 23:51:18.273708 18046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d4d4e31823e410f95781c3cb97da809 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4d4e31823e410f95781c3cb97da809" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 43387 } }
I20260928 23:51:18.275143 18040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d4d4e31823e410f95781c3cb97da809 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:51:18.275549 18048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d4d4e31823e410f95781c3cb97da809 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d4d4e31823e410f95781c3cb97da809. Latest consensus state: current_term: 1 leader_uuid: "5d4d4e31823e410f95781c3cb97da809" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4d4e31823e410f95781c3cb97da809" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 43387 } } }
I20260928 23:51:18.275664 18048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d4d4e31823e410f95781c3cb97da809 [sys.catalog]: This master's current role is: LEADER
I20260928 23:51:18.275911 18047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d4d4e31823e410f95781c3cb97da809 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d4d4e31823e410f95781c3cb97da809" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4d4e31823e410f95781c3cb97da809" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 43387 } } }
I20260928 23:51:18.276022 18047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d4d4e31823e410f95781c3cb97da809 [sys.catalog]: This master's current role is: LEADER
I20260928 23:51:18.276489 18050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:51:18.279501 18050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:51:18.279989 18040 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 23:51:18.289762 18050 catalog_manager.cc:1347] Generated new cluster ID: cb2b5fc5bbae4417b6fd02534977c2c6
I20260928 23:51:18.289913 18050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:51:18.304220 18050 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:51:18.305665 18050 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:51:18.311748 18050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d4d4e31823e410f95781c3cb97da809: Generated new TSK 0
I20260928 23:51:18.312465 18050 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
I20260928 23:51:19.061224 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.152.20:57489/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:43387
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:51:19.225200 18086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:51:19.225574 18086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:51:19.225692 18086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:51:19.230966 18086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:51:19.231114 18086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:51:19.231251 18086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:51:19.240131 18086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.152.20:57489/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:43387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:51:19.241670 18086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:51:19.243062 18086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:51:19.261204 18093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:51:19.261726 18086 server_base.cc:1034] running on GCE node
W20260928 23:51:19.262072 18096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:51:19.263803 18094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:51:19.264385 18086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:51:19.265246 18086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:51:19.272606 18086 hybrid_clock.cc:648] HybridClock initialized: now 1790639479272565 us; error 61 us; skew 500 ppm
Sep 28 23:51:19 dist-test-slave-p5bw krb5kdc[17525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639479, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:51:19.276098 18086 init.cc:377] Logged in from keytab as kudu/127.6.152.1@KRBTEST.COM (short username kudu)
I20260928 23:51:19.277261 18086 webserver.cc:460] Webserver started at http://127.6.152.1:35533/ using document root <none> and password file <none>
I20260928 23:51:19.277915 18086 fs_manager.cc:359] Metadata directory not provided
I20260928 23:51:19.278012 18086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:51:19.278225 18086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:51:19 dist-test-slave-p5bw krb5kdc[17525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639479, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for HTTP/127.6.152.20@KRBTEST.COM
I20260928 23:51:19.293238 18086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "d9619f5b8648480192dadff7c6defe20"
format_stamp: "Formatted at 2026-09-28 23:51:19 on dist-test-slave-p5bw"
server_key: "6b558852cf61651aa7d5c7b0e719dbc5"
server_key_iv: "d6ac94d116dcea984c5b218ba56e280d"
server_key_version: "kuduclusterkey@0"
I20260928 23:51:19.294391 18086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "d9619f5b8648480192dadff7c6defe20"
format_stamp: "Formatted at 2026-09-28 23:51:19 on dist-test-slave-p5bw"
server_key: "6b558852cf61651aa7d5c7b0e719dbc5"
server_key_iv: "d6ac94d116dcea984c5b218ba56e280d"
server_key_version: "kuduclusterkey@0"
I20260928 23:51:19.299038 18086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 23:51:19.322053 18103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:51:19.323827 18086 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260928 23:51:19.323987 18086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "d9619f5b8648480192dadff7c6defe20"
format_stamp: "Formatted at 2026-09-28 23:51:19 on dist-test-slave-p5bw"
server_key: "6b558852cf61651aa7d5c7b0e719dbc5"
server_key_iv: "d6ac94d116dcea984c5b218ba56e280d"
server_key_version: "kuduclusterkey@0"
I20260928 23:51:19.324137 18086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:51:19.343053 18086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:51:19.346952 18086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:51:19.347172 18086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:51:19.347848 18086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:51:19.349020 18086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:51:19.349087 18086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:51:19.349152 18086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:51:19.349186 18086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:51:19.360865 18086 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:33831
I20260928 23:51:19.360976 18217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:33831 every 8 connection(s)
I20260928 23:51:19.362035 18086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:51:19.365485 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 18086
I20260928 23:51:19.365617 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
I20260928 23:51:19.365906 6752 external_mini_cluster.cc:1428] Setting key 417fa278e54b4f308dffed9acd33f1ef
WARNING: no policy specified for kudu/127.6.152.2@KRBTEST.COM; defaulting to no policy
Sep 28 23:51:19 dist-test-slave-p5bw krb5kdc[17525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639479, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.2@KRBTEST.COM" created.
I20260928 23:51:19.396342 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:51:19.364454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:47771 (local address 127.6.152.62:43387)
0928 23:51:19.364742 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 23:51:19.364747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:51:19.366279 (+ 1532us) server_negotiation.cc:407] Connection header received
0928 23:51:19.378150 (+ 11871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:51:19.378155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:51:19.378242 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:51:19.378362 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:51:19.380258 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:51:19.381165 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:51:19.383014 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:51:19.383382 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:51:19.383488 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:51:19.390327 (+ 6839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:51:19.390331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:51:19.390335 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:51:19.390369 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:51:19.393555 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:51:19.394320 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:51:19.394324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:51:19.394326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:51:19.394435 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:51:19.394883 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:51:19.394886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:51:19.394888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:51:19.395124 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:51:19.395216 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 23:51:19.395999 (+ 783us) server_negotiation.cc:299] Negotiation successful
0928 23:51:19.396142 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":96,"threads_started":1}
I20260928 23:51:19.398155 18222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:51:19.364943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:43387 (local address 127.6.152.1:47771)
0928 23:51:19.366080 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0928 23:51:19.366145 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 23:51:19.377794 (+ 11649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:51:19.378765 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:51:19.378778 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:51:19.379207 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:51:19.380029 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:51:19.380053 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:51:19.381351 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:51:19.381355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:51:19.382839 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:51:19.382852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:51:19.383689 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:51:19.383693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:51:19.383910 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:51:19.384657 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:51:19.384679 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:51:19.390045 (+ 5366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:51:19.393720 (+ 3675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:51:19.393728 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:51:19.393750 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:51:19.394157 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:51:19.394548 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:51:19.394551 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:51:19.394554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:51:19.394770 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:51:19.395289 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:51:19.395297 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:51:19.395689 (+ 392us) client_negotiation.cc:769] Sending connection context
0928 23:51:19.396607 (+ 918us) client_negotiation.cc:240] Negotiation successful
0928 23:51:19.397029 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"thread_start_us":134,"threads_started":1}
I20260928 23:51:19.399742 18219 heartbeater.cc:347] Connected to a master server at 127.6.152.62:43387
I20260928 23:51:19.400012 18219 heartbeater.cc:464] Registering TS with master...
I20260928 23:51:19.400619 18219 heartbeater.cc:511] Master 127.6.152.62:43387 requested a full tablet report, sending...
I20260928 23:51:19.402464 18004 ts_manager.cc:194] Registered new tserver with Master: d9619f5b8648480192dadff7c6defe20 (127.6.152.1:33831)
I20260928 23:51:19.404026 18004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.1@KRBTEST.COM'} at 127.6.152.1:47771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
I20260928 23:51:19.416651 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.152.20:57489/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:43387
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:51:19 dist-test-slave-p5bw krb5kdc[17525](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790639479, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 23:51:19.556138 18225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:51:19.556404 18225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:51:19.556452 18225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:51:19.561345 18225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:51:19.561411 18225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:51:19.561506 18225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:51:19.567233 18225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.152.20:57489/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:43387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:51:19.568478 18225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:51:19.569564 18225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:51:19.579772 18231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:51:19.579838 18234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:51:19.579900 18232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:51:19.581070 18225 server_base.cc:1034] running on GCE node
I20260928 23:51:19.581465 18225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:51:19.582173 18225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:51:19.583374 18225 hybrid_clock.cc:648] HybridClock initialized: now 1790639479583343 us; error 44 us; skew 500 ppm
Sep 28 23:51:19 dist-test-slave-p5bw krb5kdc[17525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639479, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:51:19.586187 18225 init.cc:377] Logged in from keytab as kudu/127.6.152.2@KRBTEST.COM (short username kudu)
I20260928 23:51:19.587083 18225 webserver.cc:460] Webserver started at http://127.6.152.2:37491/ using document root <none> and password file <none>
I20260928 23:51:19.587647 18225 fs_manager.cc:359] Metadata directory not provided
I20260928 23:51:19.587738 18225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:51:19.587980 18225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:51:19 dist-test-slave-p5bw krb5kdc[17525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639479, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for HTTP/127.6.152.20@KRBTEST.COM
I20260928 23:51:19.614457 18225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "802888f5c845466883de109b99032ce5"
format_stamp: "Formatted at 2026-09-28 23:51:19 on dist-test-slave-p5bw"
server_key: "27423a76e935d5e4c81e80ba250f3dc7"
server_key_iv: "e14a2c0df26c133c81f82a0f7f53e632"
server_key_version: "kuduclusterkey@0"
I20260928 23:51:19.615324 18225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "802888f5c845466883de109b99032ce5"
format_stamp: "Formatted at 2026-09-28 23:51:19 on dist-test-slave-p5bw"
server_key: "27423a76e935d5e4c81e80ba250f3dc7"
server_key_iv: "e14a2c0df26c133c81f82a0f7f53e632"
server_key_version: "kuduclusterkey@0"
I20260928 23:51:19.622263 18225 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 23:51:19.641019 18241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:51:19.642699 18225 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 23:51:19.642858 18225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "802888f5c845466883de109b99032ce5"
format_stamp: "Formatted at 2026-09-28 23:51:19 on dist-test-slave-p5bw"
server_key: "27423a76e935d5e4c81e80ba250f3dc7"
server_key_iv: "e14a2c0df26c133c81f82a0f7f53e632"
server_key_version: "kuduclusterkey@0"
I20260928 23:51:19.643034 18225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:51:19.683293 18225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:51:19.686893 18225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:51:19.687105 18225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:51:19.687851 18225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:51:19.688987 18225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:51:19.689059 18225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:51:19.689137 18225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:51:19.689174 18225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:51:19.701285 18225 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:42297
I20260928 23:51:19.701455 18356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:42297 every 8 connection(s)
I20260928 23:51:19.702513 18225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:51:19.707206 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 18225
I20260928 23:51:19.707345 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:51:19.707716 6752 external_mini_cluster.cc:1428] Setting key 0d68105cc31fffcee234aa900f2517ed
Sep 28 23:51:19 dist-test-slave-p5bw krb5kdc[17525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639479, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:51:19.722537 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:51:19.708460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:38991 (local address 127.6.152.62:43387)
0928 23:51:19.708595 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 23:51:19.708600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:51:19.709081 (+ 481us) server_negotiation.cc:407] Connection header received
0928 23:51:19.710101 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:51:19.710104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:51:19.710156 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:51:19.710231 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:51:19.711844 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:51:19.712709 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:51:19.714527 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:51:19.714897 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:51:19.714998 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:51:19.718136 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:51:19.718140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:51:19.718143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:51:19.718172 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:51:19.719957 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:51:19.720586 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:51:19.720589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:51:19.720591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:51:19.720678 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:51:19.721136 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:51:19.721139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:51:19.721141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:51:19.721343 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:51:19.721437 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 23:51:19.722131 (+ 694us) server_negotiation.cc:299] Negotiation successful
0928 23:51:19.722275 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 23:51:19.723969 18359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:51:19.707075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:43387 (local address 127.6.152.2:38991)
0928 23:51:19.708927 (+ 1852us) negotiation.cc:107] Waiting for socket to connect
0928 23:51:19.708988 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 23:51:19.709888 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:51:19.710565 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:51:19.710575 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:51:19.710952 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:51:19.711656 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:51:19.711678 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:51:19.712860 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:51:19.712864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:51:19.714341 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:51:19.714364 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:51:19.715168 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:51:19.715173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:51:19.715370 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:51:19.716179 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:51:19.716199 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:51:19.717940 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:51:19.720092 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:51:19.720098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:51:19.720113 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:51:19.720448 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:51:19.720796 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:51:19.720800 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:51:19.720802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:51:19.721031 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:51:19.721467 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:51:19.721473 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:51:19.721832 (+ 359us) client_negotiation.cc:769] Sending connection context
0928 23:51:19.722051 (+ 219us) client_negotiation.cc:240] Negotiation successful
0928 23:51:19.722405 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1600,"thread_start_us":142,"threads_started":1}
I20260928 23:51:19.726720 18357 heartbeater.cc:347] Connected to a master server at 127.6.152.62:43387
WARNING: no policy specified for kudu/127.6.152.3@KRBTEST.COM; defaulting to no policy
I20260928 23:51:19.726990 18357 heartbeater.cc:464] Registering TS with master...
I20260928 23:51:19.727612 18357 heartbeater.cc:511] Master 127.6.152.62:43387 requested a full tablet report, sending...
I20260928 23:51:19.728880 18004 ts_manager.cc:194] Registered new tserver with Master: 802888f5c845466883de109b99032ce5 (127.6.152.2:42297)
I20260928 23:51:19.729588 18004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.2@KRBTEST.COM'} at 127.6.152.2:38991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
I20260928 23:51:19.746227 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.152.20:57489/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:43387
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:51:19 dist-test-slave-p5bw krb5kdc[17525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639479, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for HTTP/127.6.152.20@KRBTEST.COM
W20260928 23:51:19.901808 18362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:51:19.902135 18362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:51:19.902205 18362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:51:19.908231 18362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:51:19.908310 18362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:51:19.908394 18362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:51:19.916149 18362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.152.20:57489/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:43387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:51:19.917771 18362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:51:19.919166 18362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:51:19.932528 18367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:51:19.932754 18368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:51:19.933197 18370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:51:19.934442 18362 server_base.cc:1034] running on GCE node
I20260928 23:51:19.934914 18362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:51:19.935678 18362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:51:19.936869 18362 hybrid_clock.cc:648] HybridClock initialized: now 1790639479936794 us; error 85 us; skew 500 ppm
Sep 28 23:51:19 dist-test-slave-p5bw krb5kdc[17525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639479, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:51:19.940397 18362 init.cc:377] Logged in from keytab as kudu/127.6.152.3@KRBTEST.COM (short username kudu)
I20260928 23:51:19.941479 18362 webserver.cc:460] Webserver started at http://127.6.152.3:35267/ using document root <none> and password file <none>
I20260928 23:51:19.942211 18362 fs_manager.cc:359] Metadata directory not provided
I20260928 23:51:19.942286 18362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:51:19.942509 18362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:51:19 dist-test-slave-p5bw krb5kdc[17525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639479, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for HTTP/127.6.152.20@KRBTEST.COM
I20260928 23:51:19.977839 18362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "00087ceff4b34bb48be6f89e80a98d60"
format_stamp: "Formatted at 2026-09-28 23:51:19 on dist-test-slave-p5bw"
server_key: "f5433cbf15fa12e80d07b7b82270887c"
server_key_iv: "40d7f6938bdb854a8c7dd0c0983b2b19"
server_key_version: "kuduclusterkey@0"
I20260928 23:51:19.979492 18362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "00087ceff4b34bb48be6f89e80a98d60"
format_stamp: "Formatted at 2026-09-28 23:51:19 on dist-test-slave-p5bw"
server_key: "f5433cbf15fa12e80d07b7b82270887c"
server_key_iv: "40d7f6938bdb854a8c7dd0c0983b2b19"
server_key_version: "kuduclusterkey@0"
I20260928 23:51:19.994498 18362 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.004s sys 0.000s
I20260928 23:51:20.021240 18377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:51:20.023118 18362 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.000s
I20260928 23:51:20.023329 18362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "00087ceff4b34bb48be6f89e80a98d60"
format_stamp: "Formatted at 2026-09-28 23:51:19 on dist-test-slave-p5bw"
server_key: "f5433cbf15fa12e80d07b7b82270887c"
server_key_iv: "40d7f6938bdb854a8c7dd0c0983b2b19"
server_key_version: "kuduclusterkey@0"
I20260928 23:51:20.023543 18362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:51:20.073896 18362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:51:20.088217 18362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:51:20.088415 18362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:51:20.089120 18362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:51:20.090106 18362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:51:20.090153 18362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:51:20.090204 18362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:51:20.090224 18362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:51:20.104729 18362 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:41475
I20260928 23:51:20.106062 18362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:51:20.109302 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 18362
I20260928 23:51:20.109450 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:51:20.109803 6752 external_mini_cluster.cc:1428] Setting key df6916953fd038c2272d9d92085aa256
I20260928 23:51:20.114185 18491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:41475 every 8 connection(s)
Sep 28 23:51:20 dist-test-slave-p5bw krb5kdc[17525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639479, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:51:20.143966 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:51:20.116991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:43539 (local address 127.6.152.62:43387)
0928 23:51:20.117123 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:51:20.117129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:51:20.122457 (+ 5328us) server_negotiation.cc:407] Connection header received
0928 23:51:20.124934 (+ 2477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:51:20.124938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:51:20.124997 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:51:20.125055 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:51:20.128147 (+ 3092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.128985 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.132912 (+ 3927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.133256 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.133335 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:51:20.139446 (+ 6111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:51:20.139450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:51:20.139453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:51:20.139483 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:51:20.141351 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:51:20.142039 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:51:20.142045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:51:20.142048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:51:20.142152 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:51:20.142594 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:51:20.142597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:51:20.142599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:51:20.142835 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:51:20.142931 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 23:51:20.143667 (+ 736us) server_negotiation.cc:299] Negotiation successful
0928 23:51:20.143801 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 23:51:20.145836 18494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:51:20.116668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:43387 (local address 127.6.152.3:43539)
0928 23:51:20.122294 (+ 5626us) negotiation.cc:107] Waiting for socket to connect
0928 23:51:20.122352 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 23:51:20.123472 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:51:20.125504 (+ 2032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:51:20.125514 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:51:20.125893 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:51:20.126595 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:51:20.126615 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.129225 (+ 2610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.129228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:51:20.130628 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:51:20.130640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.133541 (+ 2901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.133546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:51:20.133755 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:51:20.137256 (+ 3501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:51:20.137279 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:51:20.139238 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:51:20.141484 (+ 2246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:51:20.141491 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:51:20.141509 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:51:20.141894 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:51:20.142281 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:51:20.142287 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:51:20.142291 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:51:20.142480 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:51:20.143026 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:51:20.143033 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:51:20.143362 (+ 329us) client_negotiation.cc:769] Sending connection context
0928 23:51:20.144434 (+ 1072us) client_negotiation.cc:240] Negotiation successful
0928 23:51:20.144794 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5381,"thread_start_us":181,"threads_started":1}
I20260928 23:51:20.158253 18492 heartbeater.cc:347] Connected to a master server at 127.6.152.62:43387
I20260928 23:51:20.158537 18492 heartbeater.cc:464] Registering TS with master...
I20260928 23:51:20.159170 18492 heartbeater.cc:511] Master 127.6.152.62:43387 requested a full tablet report, sending...
I20260928 23:51:20.160801 18004 ts_manager.cc:194] Registered new tserver with Master: 00087ceff4b34bb48be6f89e80a98d60 (127.6.152.3:41475)
I20260928 23:51:20.161582 18004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.3@KRBTEST.COM'} at 127.6.152.3:43539
I20260928 23:51:20.162889 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:51:20.198946 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:51:20.173273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48532 (local address 127.6.152.62:43387)
0928 23:51:20.173397 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 23:51:20.173402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:51:20.189236 (+ 15834us) server_negotiation.cc:407] Connection header received
0928 23:51:20.189485 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:51:20.189489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:51:20.189548 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:51:20.189606 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:51:20.190732 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.191556 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.192881 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.193215 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.193306 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:51:20.195202 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:51:20.195206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:51:20.195209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:51:20.195236 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:51:20.197023 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:51:20.197544 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:51:20.197548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:51:20.197550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:51:20.197641 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:51:20.197958 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:51:20.197961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:51:20.197963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:51:20.198189 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:51:20.198293 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 23:51:20.198654 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 23:51:20.198775 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:51:20.203529 18004 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:51:20.208212 18004 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:51:20.250897 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:51:20.241068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:41475 (local address 127.0.0.1:52708)
0928 23:51:20.242356 (+ 1288us) negotiation.cc:107] Waiting for socket to connect
0928 23:51:20.242409 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 23:51:20.242554 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:51:20.245857 (+ 3303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:51:20.245863 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:51:20.245889 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:51:20.246022 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:51:20.246029 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.247516 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.247520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:51:20.248943 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:51:20.248955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.250350 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.250354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:51:20.250465 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:51:20.250516 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 23:51:20.250626 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 23:51:20.250745 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1083,"thread_start_us":108,"threads_started":1}
I20260928 23:51:20.251947 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:51:20.241180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52708 (local address 127.6.152.3:41475)
0928 23:51:20.245156 (+ 3976us) server_negotiation.cc:206] Beginning negotiation
0928 23:51:20.245168 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:51:20.245193 (+ 25us) server_negotiation.cc:407] Connection header received
0928 23:51:20.245276 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:51:20.245282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:51:20.245516 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:51:20.245676 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:51:20.246161 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.247368 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.249113 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.250028 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.250130 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:51:20.250290 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 23:51:20.251703 (+ 1413us) server_negotiation.cc:299] Negotiation successful
0928 23:51:20.251808 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3856,"thread_start_us":81,"threads_started":1}
I20260928 23:51:20.255400 18412 tablet_service.cc:1467] Processing CreateTablet for tablet fa809db0c8c54b2b8f37f1134d5c9fe7 (DEFAULT_TABLE table=test-table [id=ace458e3053d4ab682333071294006b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:51:20.256788 18412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa809db0c8c54b2b8f37f1134d5c9fe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:51:20.268093 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:51:20.249151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:33831 (local address 127.0.0.1:37420)
0928 23:51:20.251494 (+ 2343us) negotiation.cc:107] Waiting for socket to connect
0928 23:51:20.251523 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 23:51:20.251587 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:51:20.260437 (+ 8850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:51:20.260440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:51:20.260457 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:51:20.260528 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:51:20.260533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.265368 (+ 4835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.265372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:51:20.266506 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:51:20.266516 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.267726 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.267729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:51:20.267812 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:51:20.267834 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:51:20.267895 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 23:51:20.267959 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2286,"thread_start_us":111,"threads_started":1}
I20260928 23:51:20.268561 18508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:51:20.249498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:42297 (local address 127.0.0.1:49628)
0928 23:51:20.257181 (+ 7683us) negotiation.cc:107] Waiting for socket to connect
0928 23:51:20.257195 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:51:20.257294 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:51:20.258625 (+ 1331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:51:20.258628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:51:20.258646 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:51:20.258751 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:51:20.258757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.262046 (+ 3289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.262050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:51:20.263308 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:51:20.263318 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.264547 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.264551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:51:20.264653 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:51:20.264676 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:51:20.264741 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 23:51:20.264827 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7621,"thread_start_us":69,"threads_started":1}
I20260928 23:51:20.269202 18511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:51:20.249795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49628 (local address 127.6.152.2:42297)
0928 23:51:20.257025 (+ 7230us) server_negotiation.cc:206] Beginning negotiation
0928 23:51:20.257032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:51:20.257395 (+ 363us) server_negotiation.cc:407] Connection header received
0928 23:51:20.257478 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:51:20.257484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:51:20.257692 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:51:20.257835 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:51:20.258880 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.261942 (+ 3062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.263441 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.264250 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.264353 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:51:20.264492 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 23:51:20.268951 (+ 4459us) server_negotiation.cc:299] Negotiation successful
0928 23:51:20.269054 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7114,"thread_start_us":82,"threads_started":1}
I20260928 23:51:20.272280 18290 tablet_service.cc:1467] Processing CreateTablet for tablet fa809db0c8c54b2b8f37f1134d5c9fe7 (DEFAULT_TABLE table=test-table [id=ace458e3053d4ab682333071294006b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:51:20.273241 18512 tablet_bootstrap.cc:492] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60: Bootstrap starting.
I20260928 23:51:20.273411 18290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa809db0c8c54b2b8f37f1134d5c9fe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:51:20.276243 18512 tablet_bootstrap.cc:654] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60: Neither blocks nor log segments found. Creating new log.
I20260928 23:51:20.285807 18512 log.cc:824] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60: Log is configured to *not* fsync() on all Append() calls
I20260928 23:51:20.286190 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:51:20.249310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37420 (local address 127.6.152.1:33831)
0928 23:51:20.258149 (+ 8839us) server_negotiation.cc:206] Beginning negotiation
0928 23:51:20.258158 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:51:20.258182 (+ 24us) server_negotiation.cc:407] Connection header received
0928 23:51:20.258255 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:51:20.258260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:51:20.260144 (+ 1884us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:51:20.260295 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:51:20.260655 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.261795 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.266654 (+ 4859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.267455 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.267549 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:51:20.267670 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 23:51:20.285874 (+ 18204us) server_negotiation.cc:299] Negotiation successful
0928 23:51:20.286030 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8712,"thread_start_us":128,"threads_started":1}
I20260928 23:51:20.289829 18144 tablet_service.cc:1467] Processing CreateTablet for tablet fa809db0c8c54b2b8f37f1134d5c9fe7 (DEFAULT_TABLE table=test-table [id=ace458e3053d4ab682333071294006b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:51:20.290961 18144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa809db0c8c54b2b8f37f1134d5c9fe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:51:20.294800 18512 tablet_bootstrap.cc:492] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60: No bootstrap required, opened a new log
I20260928 23:51:20.295095 18512 ts_tablet_manager.cc:1397] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20260928 23:51:20.299481 18512 raft_consensus.cc:348] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9619f5b8648480192dadff7c6defe20" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33831 } } peers { permanent_uuid: "00087ceff4b34bb48be6f89e80a98d60" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 41475 } } peers { permanent_uuid: "802888f5c845466883de109b99032ce5" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 42297 } }
I20260928 23:51:20.299930 18512 raft_consensus.cc:374] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:51:20.300030 18512 raft_consensus.cc:729] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00087ceff4b34bb48be6f89e80a98d60, State: Initialized, Role: FOLLOWER
I20260928 23:51:20.300623 18512 consensus_queue.cc:260] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9619f5b8648480192dadff7c6defe20" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33831 } } peers { permanent_uuid: "00087ceff4b34bb48be6f89e80a98d60" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 41475 } } peers { permanent_uuid: "802888f5c845466883de109b99032ce5" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 42297 } }
I20260928 23:51:20.301817 18512 ts_tablet_manager.cc:1428] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260928 23:51:20.301878 18514 tablet_bootstrap.cc:492] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 802888f5c845466883de109b99032ce5: Bootstrap starting.
I20260928 23:51:20.302218 18492 heartbeater.cc:503] Master 127.6.152.62:43387 was elected leader, sending a full tablet report...
I20260928 23:51:20.304631 18514 tablet_bootstrap.cc:654] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 802888f5c845466883de109b99032ce5: Neither blocks nor log segments found. Creating new log.
I20260928 23:51:20.305578 18514 log.cc:824] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 802888f5c845466883de109b99032ce5: Log is configured to *not* fsync() on all Append() calls
I20260928 23:51:20.310536 18515 tablet_bootstrap.cc:492] T fa809db0c8c54b2b8f37f1134d5c9fe7 P d9619f5b8648480192dadff7c6defe20: Bootstrap starting.
Sep 28 23:51:20 dist-test-slave-p5bw krb5kdc[17525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639479, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for HTTP/127.6.152.20@KRBTEST.COM
I20260928 23:51:20.311172 18514 tablet_bootstrap.cc:492] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 802888f5c845466883de109b99032ce5: No bootstrap required, opened a new log
I20260928 23:51:20.311430 18514 ts_tablet_manager.cc:1397] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 802888f5c845466883de109b99032ce5: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260928 23:51:20.313246 18515 tablet_bootstrap.cc:654] T fa809db0c8c54b2b8f37f1134d5c9fe7 P d9619f5b8648480192dadff7c6defe20: Neither blocks nor log segments found. Creating new log.
I20260928 23:51:20.316596 18514 raft_consensus.cc:348] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 802888f5c845466883de109b99032ce5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9619f5b8648480192dadff7c6defe20" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33831 } } peers { permanent_uuid: "00087ceff4b34bb48be6f89e80a98d60" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 41475 } } peers { permanent_uuid: "802888f5c845466883de109b99032ce5" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 42297 } }
I20260928 23:51:20.317039 18514 raft_consensus.cc:374] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 802888f5c845466883de109b99032ce5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:51:20.317155 18514 raft_consensus.cc:729] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 802888f5c845466883de109b99032ce5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 802888f5c845466883de109b99032ce5, State: Initialized, Role: FOLLOWER
I20260928 23:51:20.322335 18514 consensus_queue.cc:260] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 802888f5c845466883de109b99032ce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9619f5b8648480192dadff7c6defe20" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33831 } } peers { permanent_uuid: "00087ceff4b34bb48be6f89e80a98d60" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 41475 } } peers { permanent_uuid: "802888f5c845466883de109b99032ce5" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 42297 } }
I20260928 23:51:20.323985 18515 log.cc:824] T fa809db0c8c54b2b8f37f1134d5c9fe7 P d9619f5b8648480192dadff7c6defe20: Log is configured to *not* fsync() on all Append() calls
I20260928 23:51:20.326921 18515 tablet_bootstrap.cc:492] T fa809db0c8c54b2b8f37f1134d5c9fe7 P d9619f5b8648480192dadff7c6defe20: No bootstrap required, opened a new log
I20260928 23:51:20.327160 18515 ts_tablet_manager.cc:1397] T fa809db0c8c54b2b8f37f1134d5c9fe7 P d9619f5b8648480192dadff7c6defe20: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20260928 23:51:20.333285 18515 raft_consensus.cc:348] T fa809db0c8c54b2b8f37f1134d5c9fe7 P d9619f5b8648480192dadff7c6defe20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9619f5b8648480192dadff7c6defe20" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33831 } } peers { permanent_uuid: "00087ceff4b34bb48be6f89e80a98d60" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 41475 } } peers { permanent_uuid: "802888f5c845466883de109b99032ce5" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 42297 } }
I20260928 23:51:20.333776 18515 raft_consensus.cc:374] T fa809db0c8c54b2b8f37f1134d5c9fe7 P d9619f5b8648480192dadff7c6defe20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:51:20.333885 18515 raft_consensus.cc:729] T fa809db0c8c54b2b8f37f1134d5c9fe7 P d9619f5b8648480192dadff7c6defe20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9619f5b8648480192dadff7c6defe20, State: Initialized, Role: FOLLOWER
I20260928 23:51:20.334432 18515 consensus_queue.cc:260] T fa809db0c8c54b2b8f37f1134d5c9fe7 P d9619f5b8648480192dadff7c6defe20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9619f5b8648480192dadff7c6defe20" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33831 } } peers { permanent_uuid: "00087ceff4b34bb48be6f89e80a98d60" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 41475 } } peers { permanent_uuid: "802888f5c845466883de109b99032ce5" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 42297 } }
I20260928 23:51:20.335582 18515 ts_tablet_manager.cc:1428] T fa809db0c8c54b2b8f37f1134d5c9fe7 P d9619f5b8648480192dadff7c6defe20: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260928 23:51:20.336004 18219 heartbeater.cc:503] Master 127.6.152.62:43387 was elected leader, sending a full tablet report...
I20260928 23:51:20.345305 18514 ts_tablet_manager.cc:1428] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 802888f5c845466883de109b99032ce5: Time spent starting tablet: real 0.034s user 0.004s sys 0.003s
I20260928 23:51:20.345831 18357 heartbeater.cc:503] Master 127.6.152.62:43387 was elected leader, sending a full tablet report...
W20260928 23:51:20.363667 18220 tablet.cc:2369] T fa809db0c8c54b2b8f37f1134d5c9fe7 P d9619f5b8648480192dadff7c6defe20: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:51:20.368557 18493 tablet.cc:2369] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:51:20.454011 18358 tablet.cc:2369] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 802888f5c845466883de109b99032ce5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:51:20.606320 18516 raft_consensus.cc:482] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:51:20.606551 18516 raft_consensus.cc:504] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9619f5b8648480192dadff7c6defe20" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33831 } } peers { permanent_uuid: "00087ceff4b34bb48be6f89e80a98d60" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 41475 } } peers { permanent_uuid: "802888f5c845466883de109b99032ce5" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 42297 } }
I20260928 23:51:20.607951 18516 leader_election.cc:288] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9619f5b8648480192dadff7c6defe20 (127.6.152.1:33831), 802888f5c845466883de109b99032ce5 (127.6.152.2:42297)
I20260928 23:51:20.614022 18494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:51:20.608490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:33831 (local address 127.6.152.3:51157)
0928 23:51:20.609141 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0928 23:51:20.609158 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:51:20.609255 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:51:20.610130 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:51:20.610134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:51:20.610155 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:51:20.610276 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:51:20.610284 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.611358 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.611362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:51:20.612591 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:51:20.612602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.613607 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.613611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:51:20.613716 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:51:20.613738 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:51:20.613808 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 23:51:20.613879 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563}
I20260928 23:51:20.615301 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:51:20.609512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:51157 (local address 127.6.152.1:33831)
0928 23:51:20.609663 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 23:51:20.609668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:51:20.609686 (+ 18us) server_negotiation.cc:407] Connection header received
0928 23:51:20.609744 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:51:20.609748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:51:20.609799 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:51:20.609885 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:51:20.610423 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.611205 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.612740 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.613378 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.613481 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:51:20.613549 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 23:51:20.615085 (+ 1536us) server_negotiation.cc:299] Negotiation successful
0928 23:51:20.615181 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:51:20.615962 18162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa809db0c8c54b2b8f37f1134d5c9fe7" candidate_uuid: "00087ceff4b34bb48be6f89e80a98d60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9619f5b8648480192dadff7c6defe20" is_pre_election: true
I20260928 23:51:20.616334 18162 raft_consensus.cc:2454] T fa809db0c8c54b2b8f37f1134d5c9fe7 P d9619f5b8648480192dadff7c6defe20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00087ceff4b34bb48be6f89e80a98d60 in term 0.
I20260928 23:51:20.617162 18381 leader_election.cc:302] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00087ceff4b34bb48be6f89e80a98d60, d9619f5b8648480192dadff7c6defe20; no voters:
I20260928 23:51:20.617493 18516 raft_consensus.cc:2781] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:51:20.617573 18516 raft_consensus.cc:482] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:51:20.617617 18516 raft_consensus.cc:3037] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:51:20.619028 18516 raft_consensus.cc:504] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9619f5b8648480192dadff7c6defe20" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33831 } } peers { permanent_uuid: "00087ceff4b34bb48be6f89e80a98d60" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 41475 } } peers { permanent_uuid: "802888f5c845466883de109b99032ce5" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 42297 } }
I20260928 23:51:20.619400 18516 leader_election.cc:288] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60 [CANDIDATE]: Term 1 election: Requested vote from peers d9619f5b8648480192dadff7c6defe20 (127.6.152.1:33831), 802888f5c845466883de109b99032ce5 (127.6.152.2:42297)
I20260928 23:51:20.621066 18162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa809db0c8c54b2b8f37f1134d5c9fe7" candidate_uuid: "00087ceff4b34bb48be6f89e80a98d60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9619f5b8648480192dadff7c6defe20"
I20260928 23:51:20.621227 18162 raft_consensus.cc:3037] T fa809db0c8c54b2b8f37f1134d5c9fe7 P d9619f5b8648480192dadff7c6defe20 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:51:20.622493 18162 raft_consensus.cc:2454] T fa809db0c8c54b2b8f37f1134d5c9fe7 P d9619f5b8648480192dadff7c6defe20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00087ceff4b34bb48be6f89e80a98d60 in term 1.
I20260928 23:51:20.622911 18381 leader_election.cc:302] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00087ceff4b34bb48be6f89e80a98d60, d9619f5b8648480192dadff7c6defe20; no voters:
I20260928 23:51:20.623508 18516 raft_consensus.cc:2781] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:51:20.623734 18516 raft_consensus.cc:686] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60 [term 1 LEADER]: Becoming Leader. State: Replica: 00087ceff4b34bb48be6f89e80a98d60, State: Running, Role: LEADER
I20260928 23:51:20.624122 18516 consensus_queue.cc:237] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9619f5b8648480192dadff7c6defe20" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33831 } } peers { permanent_uuid: "00087ceff4b34bb48be6f89e80a98d60" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 41475 } } peers { permanent_uuid: "802888f5c845466883de109b99032ce5" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 42297 } }
I20260928 23:51:20.628980 18494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:51:20.608729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:42297 (local address 127.6.152.3:58627)
0928 23:51:20.614208 (+ 5479us) negotiation.cc:107] Waiting for socket to connect
0928 23:51:20.614237 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 23:51:20.614317 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:51:20.615947 (+ 1630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:51:20.615950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:51:20.615969 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:51:20.616063 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:51:20.616069 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.624965 (+ 8896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.624969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:51:20.626163 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:51:20.626176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.628530 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.628534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:51:20.628629 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:51:20.628650 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 23:51:20.628717 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 23:51:20.628793 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5427,"thread_start_us":107,"threads_started":1}
I20260928 23:51:20.629455 18511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:51:20.609669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:58627 (local address 127.6.152.2:42297)
0928 23:51:20.610058 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0928 23:51:20.610064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:51:20.614778 (+ 4714us) server_negotiation.cc:407] Connection header received
0928 23:51:20.614842 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:51:20.614845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:51:20.614900 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:51:20.614975 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:51:20.616279 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.623352 (+ 7073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.627560 (+ 4208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.628232 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.628346 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:51:20.628413 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 23:51:20.629246 (+ 833us) server_negotiation.cc:299] Negotiation successful
0928 23:51:20.629333 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304}
I20260928 23:51:20.629099 18004 catalog_manager.cc:5495] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00087ceff4b34bb48be6f89e80a98d60 (127.6.152.3). New cstate: current_term: 1 leader_uuid: "00087ceff4b34bb48be6f89e80a98d60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9619f5b8648480192dadff7c6defe20" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00087ceff4b34bb48be6f89e80a98d60" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 41475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "802888f5c845466883de109b99032ce5" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 42297 } health_report { overall_health: UNKNOWN } } }
I20260928 23:51:20.630610 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa809db0c8c54b2b8f37f1134d5c9fe7" candidate_uuid: "00087ceff4b34bb48be6f89e80a98d60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "802888f5c845466883de109b99032ce5" is_pre_election: true
I20260928 23:51:20.630892 18310 raft_consensus.cc:2454] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 802888f5c845466883de109b99032ce5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00087ceff4b34bb48be6f89e80a98d60 in term 0.
I20260928 23:51:20.631219 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa809db0c8c54b2b8f37f1134d5c9fe7" candidate_uuid: "00087ceff4b34bb48be6f89e80a98d60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "802888f5c845466883de109b99032ce5"
I20260928 23:51:20.631316 18309 raft_consensus.cc:3037] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 802888f5c845466883de109b99032ce5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:51:20.632611 18309 raft_consensus.cc:2454] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 802888f5c845466883de109b99032ce5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00087ceff4b34bb48be6f89e80a98d60 in term 1.
I20260928 23:51:20.718142 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:51:20.713672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52722 (local address 127.6.152.3:41475)
0928 23:51:20.713803 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 23:51:20.713808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:51:20.713943 (+ 135us) server_negotiation.cc:407] Connection header received
0928 23:51:20.714159 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:51:20.714163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:51:20.714249 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:51:20.714370 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:51:20.714736 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.715531 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.716387 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:51:20.716726 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:51:20.716863 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:51:20.717180 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:51:20.717740 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:51:20.717863 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 23:51:20.717904 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 23:51:20.717967 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:51:20.723860 18162 raft_consensus.cc:1261] T fa809db0c8c54b2b8f37f1134d5c9fe7 P d9619f5b8648480192dadff7c6defe20 [term 1 FOLLOWER]: Refusing update from remote peer 00087ceff4b34bb48be6f89e80a98d60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:51:20.723860 18309 raft_consensus.cc:1261] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 802888f5c845466883de109b99032ce5 [term 1 FOLLOWER]: Refusing update from remote peer 00087ceff4b34bb48be6f89e80a98d60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:51:20.724573 18523 consensus_queue.cc:1035] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "802888f5c845466883de109b99032ce5" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 42297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:51:20.724869 18516 consensus_queue.cc:1035] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9619f5b8648480192dadff7c6defe20" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:51:20.733973 18534 mvcc.cc:205] Tried to move back new op lower bound from 7334459313037578240 to 7334459312645677056. Current Snapshot: MvccSnapshot[applied={T|T < 7334459313037578240}]
I20260928 23:51:20.734980 18535 mvcc.cc:205] Tried to move back new op lower bound from 7334459313037578240 to 7334459312645677056. Current Snapshot: MvccSnapshot[applied={T|T < 7334459313037578240}]
I20260928 23:51:20.735965 18536 mvcc.cc:205] Tried to move back new op lower bound from 7334459313037578240 to 7334459312645677056. Current Snapshot: MvccSnapshot[applied={T|T < 7334459313037578240}]
I20260928 23:51:20.745129 18004 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:51:20.745319 18004 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:51:20.747861 18004 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d4d4e31823e410f95781c3cb97da809: Sending DeleteTablet for 3 replicas of tablet fa809db0c8c54b2b8f37f1134d5c9fe7
I20260928 23:51:20.748749 18144 tablet_service.cc:1514] Processing DeleteTablet for tablet fa809db0c8c54b2b8f37f1134d5c9fe7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:51:20 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:37420
I20260928 23:51:20.748914 18290 tablet_service.cc:1514] Processing DeleteTablet for tablet fa809db0c8c54b2b8f37f1134d5c9fe7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:51:20 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:49628
I20260928 23:51:20.749416 18544 tablet_replica.cc:330] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 802888f5c845466883de109b99032ce5: stopping tablet replica
I20260928 23:51:20.749735 18544 raft_consensus.cc:2229] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 802888f5c845466883de109b99032ce5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:51:20.749780 18412 tablet_service.cc:1514] Processing DeleteTablet for tablet fa809db0c8c54b2b8f37f1134d5c9fe7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:51:20 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:52708
I20260928 23:51:20.750067 18544 raft_consensus.cc:2258] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 802888f5c845466883de109b99032ce5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:51:20.750429 18545 tablet_replica.cc:330] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60: stopping tablet replica
I20260928 23:51:20.750712 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 18086
I20260928 23:51:20.751396 18545 raft_consensus.cc:2229] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:51:20.751775 18545 raft_consensus.cc:2258] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:51:20.753202 18545 ts_tablet_manager.cc:1905] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:51:20.755853 18545 ts_tablet_manager.cc:1918] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:51:20.755944 18545 log.cc:1196] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/fa809db0c8c54b2b8f37f1134d5c9fe7
I20260928 23:51:20.756250 18545 ts_tablet_manager.cc:1939] T fa809db0c8c54b2b8f37f1134d5c9fe7 P 00087ceff4b34bb48be6f89e80a98d60: Deleting consensus metadata
I20260928 23:51:20.757393 17988 catalog_manager.cc:4841] TS 00087ceff4b34bb48be6f89e80a98d60 (127.6.152.3:41475): tablet fa809db0c8c54b2b8f37f1134d5c9fe7 (table test-table [id=ace458e3053d4ab682333071294006b0]) successfully deleted
W20260928 23:51:20.758389 17991 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.152.1:33831 (error 108)
I20260928 23:51:20.759322 18508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:51:20.759044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:33831 (local address 127.0.0.1:37436)
0928 23:51:20.759177 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 23:51:20.759241 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.1:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 23:51:20.759404 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 18225
W20260928 23:51:20.760149 17991 catalog_manager.cc:4584] TS d9619f5b8648480192dadff7c6defe20 (127.6.152.1:33831): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fa809db0c8c54b2b8f37f1134d5c9fe7: Network error: Client connection negotiation failed: client connection to 127.6.152.1:33831: connect: Connection refused (error 111)
I20260928 23:51:20.765159 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 18362
I20260928 23:51:20.766582 18508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:51:20.766303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:42297 (local address 127.0.0.1:49632)
0928 23:51:20.766452 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 23:51:20.766498 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.2:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 23:51:20.770994 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 17972
I20260928 23:51:20.776351 6752 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 23:51:21.111145 6752 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 23:51:21.111303 6752 mini_ranger.cc:66] Stopping Ranger...
I20260928 23:51:21.456688 6752 mini_ranger.cc:68] Stopped Ranger
2026-09-28 23:51:21.456 UTC [17555] LOG: received smart shutdown request
2026-09-28 23:51:21.460 UTC [17555] LOG: background worker "logical replication launcher" (PID 17564) exited with exit code 1
2026-09-28 23:51:21.460 UTC [17559] LOG: shutting down
2026-09-28 23:51:21.636 UTC [17555] LOG: database system is shut down
2026-09-28T23:51:21Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46211 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:51:21 dist-test-slave-p5bw krb5kdc[18555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:51:21 dist-test-slave-p5bw krb5kdc[18555](info): set up 2 sockets
Sep 28 23:51:21 dist-test-slave-p5bw krb5kdc[18555](info): commencing operation
krb5kdc: starting...
W20260928 23:51:22.781755 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:51:22 dist-test-slave-p5bw krb5kdc[18555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:51:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:51:22Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.152.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.152.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.152.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/rangeradmin_127.6.152.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.152.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/rangeradmin_127.6.152.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.152.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.152.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.152.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/rangerlookup_127.6.152.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.152.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/rangerlookup_127.6.152.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.152.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.152.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.152.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/HTTP_127.6.152.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.152.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/HTTP_127.6.152.20@KRBTEST.COM.keytab.
I20260928 23:51:22.951098 6752 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskXX9CKr/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/postgres -l logfile start
2026-09-28 23:51:25.501 UTC [18585] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 23:51:25.501 UTC [18585] LOG: listening on IPv4 address "127.6.152.20", port 57757
2026-09-28 23:51:25.517 UTC [18585] LOG: listening on Unix socket "/tmp/.s.PGSQL.57757"
2026-09-28 23:51:25.553 UTC [18588] LOG: database system was shut down at 2026-09-28 23:51:24 UTC
2026-09-28 23:51:25.557 UTC [18585] LOG: database system is ready to accept connections
I20260928 23:51:26.481660 6752 mini_postgres.cc:96] Postgres bound to 57757
2026-09-28 23:51:26.487 UTC [18596] FATAL: database "slave" does not exist
127.6.152.20:57757 - accepting connections
I20260928 23:51:26.488539 6752 mini_ranger.cc:161] Starting Ranger...
I20260928 23:51:26.503633 6752 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 23:51:26.983014 6752 mini_ranger.cc:87] Created ranger Postgres database
I20260928 23:51:26.983168 6752 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ranger-admin
2026-09-28 23:51:27,259 [I] DB FLAVOR :POSTGRES
2026-09-28 23:51:27,259 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 23:51:27,259 [I] Checking connection..
2026-09-28 23:51:27,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:57757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 23:51:27,512 [I] Checking connection passed.
2026-09-28 23:51:27,512 [I] --------- Verifying version history table ---------
2026-09-28 23:51:27,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:57757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:51:27,740 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 23:51:27,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:57757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:51:27,966 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 23:51:27,966 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 23:51:27,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:57757/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 23:51:28.188 UTC [18661] WARNING: there is no transaction in progress
2026-09-28 23:51:28,194 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 23:51:28,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:57757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:51:28,439 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 23:51:28,439 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 23:51:28,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:57757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 23:51:28,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:57757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 23:51:28,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:57757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-p5bw.c.gcp-upstream.internal','N') ;"
2026-09-28 23:51:29,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:57757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 23:51:29,446 [I] Table x_portal_user does not exist in database ranger
2026-09-28 23:51:29,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:57757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 23:51:29,677 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 23:51:29,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:57757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 23:51:29,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:57757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 23:51:30,155 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 23:51:30,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:57757/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 23:51:31.113 UTC [18769] WARNING: there is no transaction in progress
2026-09-28 23:51:31.130 UTC [18769] WARNING: there is no transaction in progress
2026-09-28 23:51:31.146 UTC [18769] WARNING: there is no transaction in progress
2026-09-28 23:51:31.162 UTC [18769] WARNING: there is no transaction in progress
2026-09-28 23:51:31.178 UTC [18769] WARNING: there is no transaction in progress
2026-09-28 23:51:31.196 UTC [18769] WARNING: there is no transaction in progress
2026-09-28 23:51:31.251 UTC [18769] WARNING: there is no transaction in progress
2026-09-28 23:51:31.258 UTC [18769] WARNING: there is no transaction in progress
2026-09-28 23:51:31.273 UTC [18769] WARNING: there is no transaction in progress
2026-09-28 23:51:31.284 UTC [18769] WARNING: there is no transaction in progress
2026-09-28 23:51:31.295 UTC [18769] WARNING: there is no transaction in progress
2026-09-28 23:51:31.308 UTC [18769] WARNING: there is no transaction in progress
2026-09-28 23:51:31.318 UTC [18769] WARNING: there is no transaction in progress
2026-09-28 23:51:31.325 UTC [18769] WARNING: there is no transaction in progress
2026-09-28 23:51:31.333 UTC [18769] WARNING: there is no transaction in progress
2026-09-28 23:51:31.340 UTC [18769] WARNING: there is no transaction in progress
2026-09-28 23:51:31,989 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 23:51:31,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:57757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 23:51:32,219 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 23:51:32,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:57757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 23:51:32,478 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 23:51:32,478 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:57757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 23:51:32,712 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 23:51:32,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:57757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 23:51:32,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:57757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 23:51:33,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:57757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-p5bw.c.gcp-upstream.internal';"
2026-09-28 23:51:33,456 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 23:51:33,457 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:57757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 23:51:33,703 [I] DB_PATCHES have already been applied
I20260928 23:51:33.708765 6752 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ranger-admin:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskXX9CKr/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 23:51:33.708895 6752 mini_ranger.cc:193] Using host: 127.6.152.20
I20260928 23:51:33.712216 6752 mini_ranger.cc:227] Ranger admin URL: http://127.6.152.20:35365
Sep 28, 2026 11:51:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 11:51:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 11:51:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 11:51:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 11:51:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 11:51:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 11:51:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 23:51:34.827257 6752 mini_ranger.cc:160] Time spent starting Ranger: real 8.339s user 0.000s sys 0.010s
Sep 28, 2026 11:51:54 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 11:51:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 11:51:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 11:51:54 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 11:51:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 11:51:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 11:51:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 11:51:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 11:51:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 11:51:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 11:51:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 11:51:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 11:51:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 11:51:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 11:51:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 11:51:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 11:51:54 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 11:51:54 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 23:51:55.509847 6752 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.152.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.152.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.152.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/rangerkms_127.6.152.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.152.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/rangerkms_127.6.152.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.152.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.152.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.152.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/HTTP_127.6.152.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.152.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/HTTP_127.6.152.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 23:51:55 dist-test-slave-p5bw krb5kdc[18555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639515, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 23:51:55.594712 6752 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 23:51:55.603891 6752 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 23:51:56.486227 6752 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 23:51:56.486301 6752 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ranger-kms
I20260928 23:51:56.486315 6752 mini_ranger_kms.cc:227] Using postgres at 127.6.152.20:57757
2026-09-28 23:51:56,654 [I] DB FLAVOR :POSTGRES
2026-09-28 23:51:56,655 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 23:51:56,655 [I] Checking connection
2026-09-28 23:51:56,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:57757/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 23:51:56,891 [I] connection success
2026-09-28 23:51:56,892 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 23:51:56,892 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 23:51:56,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:57757/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 23:51:57,135 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 23:51:57,135 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 23:51:57,136 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 23:51:57,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.152.20:57757/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 23:51:57,397 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 23:51:57.870927 6752 mini_ranger_kms.cc:310] Created kms service
I20260928 23:51:58.096197 6752 mini_ranger_kms.cc:326] Created kudu user
I20260928 23:51:58.137622 6752 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 23:51:58 dist-test-slave-p5bw krb5kdc[18555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639515, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.152.20@KRBTEST.COM
I20260928 23:51:58.486758 6752 mini_ranger_kms.cc:384] Added ranger policy
I20260928 23:51:58.486904 6752 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ranger-kms:/tmp/dist-test-taskXX9CKr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskXX9CKr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 23:51:58.486925 6752 mini_ranger_kms.cc:240] Using host: 127.6.152.20
I20260928 23:51:58.491621 6752 mini_ranger_kms.cc:276] Ranger KMS PID: 18949
I20260928 23:51:58.491704 6752 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.152.20:38615
Sep 28, 2026 11:51:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 11:51:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 11:51:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 11:51:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 11:52:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 11:52:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 23:52:00 dist-test-slave-p5bw krb5kdc[18555](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790639520, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.152.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:00.843117 6752 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.248s user 0.007s sys 0.145s
I20260928 23:52:00.843243 6752 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 23:52:00.843312 6752 mini_ranger_kms.cc:401] 127.6.152.20:38615/kms/v1/keys
Sep 28 23:52:01 dist-test-slave-p5bw krb5kdc[18555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639520, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.152.20@KRBTEST.COM for HTTP/127.6.152.20@KRBTEST.COM
Sep 28 23:52:01 dist-test-slave-p5bw krb5kdc[18555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639520, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.152.20@KRBTEST.COM for HTTP/127.6.152.20@KRBTEST.COM
Sep 28 23:52:03 dist-test-slave-p5bw krb5kdc[18555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:52:03.720109 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:41649
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:41649
--ranger_config_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.152.20:38615/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:52:03.832274 19000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:03.832523 19000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:03.832562 19000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:03.836513 19000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:52:03.836571 19000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:03.836591 19000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:03.836607 19000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:52:03.836665 19000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:52:03.841981 19000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.152.20:38615/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:41649
--ranger_config_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:41649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:03.843034 19000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:03.844069 19000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:52:03.853055 19000 server_base.cc:1034] running on GCE node
W20260928 23:52:03.853053 19008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:03.853003 19005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:03.853220 19006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:03.853946 19000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:03.855041 19000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:03.856218 19000 hybrid_clock.cc:648] HybridClock initialized: now 1790639523856203 us; error 28 us; skew 500 ppm
Sep 28 23:52:03 dist-test-slave-p5bw krb5kdc[18555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639523, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:03.862186 19000 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:52:03.863160 19000 webserver.cc:460] Webserver started at http://127.6.152.62:33559/ using document root <none> and password file <none>
I20260928 23:52:03.863818 19000 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:03.863907 19000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:03.864126 19000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:52:03 dist-test-slave-p5bw krb5kdc[18555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639523, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for HTTP/127.6.152.20@KRBTEST.COM
I20260928 23:52:03.922811 19000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "2e2100011350430789104bff84487749"
format_stamp: "Formatted at 2026-09-28 23:52:03 on dist-test-slave-p5bw"
server_key: "8a7eefb34215daf06927fa74e3a0f676"
server_key_iv: "1dcbe52b642baf75593acb364b2a51ec"
server_key_version: "kuduclusterkey@0"
I20260928 23:52:03.923573 19000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "2e2100011350430789104bff84487749"
format_stamp: "Formatted at 2026-09-28 23:52:03 on dist-test-slave-p5bw"
server_key: "8a7eefb34215daf06927fa74e3a0f676"
server_key_iv: "1dcbe52b642baf75593acb364b2a51ec"
server_key_version: "kuduclusterkey@0"
I20260928 23:52:03.928517 19000 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 23:52:03.946117 19015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:03.947917 19000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:52:03.948073 19000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "2e2100011350430789104bff84487749"
format_stamp: "Formatted at 2026-09-28 23:52:03 on dist-test-slave-p5bw"
server_key: "8a7eefb34215daf06927fa74e3a0f676"
server_key_iv: "1dcbe52b642baf75593acb364b2a51ec"
server_key_version: "kuduclusterkey@0"
I20260928 23:52:03.948222 19000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:03.985031 19000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:03.988456 19000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:03.988662 19000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:03.997253 19000 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:41649
I20260928 23:52:03.997267 19067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:41649 every 8 connection(s)
I20260928 23:52:03.998476 19000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:52:03.999226 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 19000
I20260928 23:52:03.999353 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:52:03.999688 6752 external_mini_cluster.cc:1428] Setting key a054c599683ff0da430dd05ec98adc5c
I20260928 23:52:04.001946 19068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:52:04.008435 19068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749: Bootstrap starting.
Sep 28 23:52:04 dist-test-slave-p5bw krb5kdc[18555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:52:04.011016 19068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749: Neither blocks nor log segments found. Creating new log.
I20260928 23:52:04.012300 19068 log.cc:824] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:04.014571 19068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749: No bootstrap required, opened a new log
I20260928 23:52:04.017127 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:04.001164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49834 (local address 127.6.152.62:41649)
0928 23:52:04.001637 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:04.001658 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:04.001704 (+ 46us) server_negotiation.cc:407] Connection header received
0928 23:52:04.002601 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:04.002625 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:04.003096 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:04.003467 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:04.004374 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:04.006039 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:04.007287 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:04.008035 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:04.008210 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:04.011418 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:04.011426 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:04.011445 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:04.011481 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:04.013664 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:04.014191 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:04.014205 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:04.014211 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:04.014343 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:04.014668 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:04.014671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:04.014673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:04.015188 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:04.015370 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:04.015682 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 23:52:04.016148 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":138,"threads_started":1}
I20260928 23:52:04.017987 19068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2100011350430789104bff84487749" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41649 } }
I20260928 23:52:04.018234 19068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:52:04.018285 19068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2100011350430789104bff84487749, State: Initialized, Role: FOLLOWER
I20260928 23:52:04.018733 19068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2100011350430789104bff84487749" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41649 } }
I20260928 23:52:04.018870 19068 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:52:04.018920 19068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:52:04.019001 19068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:52:04.020059 19068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2100011350430789104bff84487749" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41649 } }
I20260928 23:52:04.020394 19068 leader_election.cc:302] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e2100011350430789104bff84487749; no voters:
I20260928 23:52:04.020714 19068 leader_election.cc:288] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:52:04.020860 19073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:52:04.021235 19073 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [term 1 LEADER]: Becoming Leader. State: Replica: 2e2100011350430789104bff84487749, State: Running, Role: LEADER
I20260928 23:52:04.021602 19073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2100011350430789104bff84487749" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41649 } }
I20260928 23:52:04.022032 19068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:52:04.022780 19073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e2100011350430789104bff84487749" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2100011350430789104bff84487749" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41649 } } }
I20260928 23:52:04.022962 19073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [sys.catalog]: This master's current role is: LEADER
I20260928 23:52:04.023212 19073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e2100011350430789104bff84487749. Latest consensus state: current_term: 1 leader_uuid: "2e2100011350430789104bff84487749" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2100011350430789104bff84487749" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41649 } } }
I20260928 23:52:04.023340 19073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [sys.catalog]: This master's current role is: LEADER
I20260928 23:52:04.023619 19082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:52:04.026839 19082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:52:04.027340 19068 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 23:52:04.035872 19082 catalog_manager.cc:1347] Generated new cluster ID: 15dfd72b1a734f4f9eb99c92e4af5ab1
I20260928 23:52:04.036242 19082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:52:04.066876 19082 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:52:04.068478 19082 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:52:04.097155 19082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749: Generated new TSK 0
I20260928 23:52:04.098352 19082 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
I20260928 23:52:04.800313 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.152.20:38615/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:41649
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:52:04.918948 19113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:04.919232 19113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:04.919306 19113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:04.923693 19113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:04.923776 19113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:04.923897 19113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:52:04.929677 19113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.152.20:38615/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:41649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:04.931018 19113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:04.932415 19113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:04.945806 19121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:04.945684 19123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:04.945684 19120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:04.946307 19113 server_base.cc:1034] running on GCE node
I20260928 23:52:04.946733 19113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:04.947427 19113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:04.948598 19113 hybrid_clock.cc:648] HybridClock initialized: now 1790639524948577 us; error 31 us; skew 500 ppm
Sep 28 23:52:04 dist-test-slave-p5bw krb5kdc[18555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639524, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:04.952311 19113 init.cc:377] Logged in from keytab as kudu/127.6.152.1@KRBTEST.COM (short username kudu)
I20260928 23:52:04.953598 19113 webserver.cc:460] Webserver started at http://127.6.152.1:36709/ using document root <none> and password file <none>
I20260928 23:52:04.954450 19113 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:04.954597 19113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:04.954898 19113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:52:04 dist-test-slave-p5bw krb5kdc[18555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639524, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for HTTP/127.6.152.20@KRBTEST.COM
I20260928 23:52:04.970738 19113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "cea9be83626d45f88737ca8c35e5ef5d"
format_stamp: "Formatted at 2026-09-28 23:52:04 on dist-test-slave-p5bw"
server_key: "1d52ec25b0a843f72f0f22cc12af222d"
server_key_iv: "45261ef3959b9c4e25aa2ac7f33b27b0"
server_key_version: "kuduclusterkey@0"
I20260928 23:52:04.971644 19113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "cea9be83626d45f88737ca8c35e5ef5d"
format_stamp: "Formatted at 2026-09-28 23:52:04 on dist-test-slave-p5bw"
server_key: "1d52ec25b0a843f72f0f22cc12af222d"
server_key_iv: "45261ef3959b9c4e25aa2ac7f33b27b0"
server_key_version: "kuduclusterkey@0"
I20260928 23:52:04.980856 19113 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.000s
I20260928 23:52:05.017297 19130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:05.019182 19113 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.003s sys 0.000s
I20260928 23:52:05.019320 19113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "cea9be83626d45f88737ca8c35e5ef5d"
format_stamp: "Formatted at 2026-09-28 23:52:04 on dist-test-slave-p5bw"
server_key: "1d52ec25b0a843f72f0f22cc12af222d"
server_key_iv: "45261ef3959b9c4e25aa2ac7f33b27b0"
server_key_version: "kuduclusterkey@0"
I20260928 23:52:05.019469 19113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:05.046518 19113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:05.050186 19113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:05.050415 19113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:05.051139 19113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:52:05.052219 19113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:52:05.052320 19113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:05.052495 19113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:52:05.052575 19113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:05.064924 19113 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:40751
I20260928 23:52:05.064960 19245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:40751 every 8 connection(s)
I20260928 23:52:05.066217 19113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:52:05.070797 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 19113
I20260928 23:52:05.070924 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
I20260928 23:52:05.071246 6752 external_mini_cluster.cc:1428] Setting key 3778c60f9a8269dd052508e638850807
Sep 28 23:52:05 dist-test-slave-p5bw krb5kdc[18555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639524, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.152.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.2@KRBTEST.COM" created.
I20260928 23:52:05.091909 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:05.069021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:44393 (local address 127.6.152.62:41649)
0928 23:52:05.077051 (+ 8030us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:05.077060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:05.077356 (+ 296us) server_negotiation.cc:407] Connection header received
0928 23:52:05.078660 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:05.078664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:05.078720 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:05.078818 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:05.080454 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:05.081253 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:05.083302 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:05.083668 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:05.083755 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:05.087068 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:05.087072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:05.087076 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:05.087105 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:05.089103 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:05.089784 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:05.089788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:05.089790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:05.089886 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:05.090413 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:05.090417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:05.090419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:05.090678 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:05.090780 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:05.091545 (+ 765us) server_negotiation.cc:299] Negotiation successful
0928 23:52:05.091703 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7938,"thread_start_us":87,"threads_started":1}
I20260928 23:52:05.093712 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:05.069188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:41649 (local address 127.6.152.1:44393)
0928 23:52:05.077161 (+ 7973us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:05.077236 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:05.078399 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:05.079149 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:05.079173 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:05.079530 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:05.080249 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:05.080271 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:05.081406 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:05.081410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:05.082866 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:05.082879 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:05.083995 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:05.083998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:05.084233 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:05.085007 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:05.085032 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:05.086852 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:05.089246 (+ 2394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:05.089254 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:05.089273 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:05.089650 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:05.090014 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:05.090017 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:05.090019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:05.090300 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:05.090859 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:05.090866 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:05.091227 (+ 361us) client_negotiation.cc:769] Sending connection context
0928 23:52:05.092153 (+ 926us) client_negotiation.cc:240] Negotiation successful
0928 23:52:05.092528 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7693,"thread_start_us":139,"threads_started":1}
I20260928 23:52:05.095211 19246 heartbeater.cc:347] Connected to a master server at 127.6.152.62:41649
I20260928 23:52:05.095480 19246 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:05.096062 19246 heartbeater.cc:511] Master 127.6.152.62:41649 requested a full tablet report, sending...
I20260928 23:52:05.097805 19032 ts_manager.cc:194] Registered new tserver with Master: cea9be83626d45f88737ca8c35e5ef5d (127.6.152.1:40751)
I20260928 23:52:05.099360 19032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.1@KRBTEST.COM'} at 127.6.152.1:44393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
I20260928 23:52:05.116232 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.152.20:38615/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:41649
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:52:05 dist-test-slave-p5bw krb5kdc[18555](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790639525, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 23:52:05.246333 19252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:05.246656 19252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:05.246767 19252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:05.251073 19252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:05.251154 19252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:05.251255 19252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:52:05.257265 19252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.152.20:38615/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:41649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:05.258464 19252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:05.259547 19252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:05.269953 19259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:05.270038 19261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:05.269935 19258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:05.271216 19252 server_base.cc:1034] running on GCE node
I20260928 23:52:05.271637 19252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:05.272379 19252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:05.273559 19252 hybrid_clock.cc:648] HybridClock initialized: now 1790639525273484 us; error 91 us; skew 500 ppm
Sep 28 23:52:05 dist-test-slave-p5bw krb5kdc[18555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639525, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:05.276998 19252 init.cc:377] Logged in from keytab as kudu/127.6.152.2@KRBTEST.COM (short username kudu)
I20260928 23:52:05.277948 19252 webserver.cc:460] Webserver started at http://127.6.152.2:38947/ using document root <none> and password file <none>
I20260928 23:52:05.278643 19252 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:05.278745 19252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:05.278987 19252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:52:05 dist-test-slave-p5bw krb5kdc[18555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639525, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for HTTP/127.6.152.20@KRBTEST.COM
I20260928 23:52:05.296034 19252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "45f06518c2b54ae391c9ce91433fd025"
format_stamp: "Formatted at 2026-09-28 23:52:05 on dist-test-slave-p5bw"
server_key: "c4bfd5a936bf5d0510069dc0a2bfee0a"
server_key_iv: "bad3b883ed9d02a210e4dd27dc790539"
server_key_version: "kuduclusterkey@0"
I20260928 23:52:05.296890 19252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "45f06518c2b54ae391c9ce91433fd025"
format_stamp: "Formatted at 2026-09-28 23:52:05 on dist-test-slave-p5bw"
server_key: "c4bfd5a936bf5d0510069dc0a2bfee0a"
server_key_iv: "bad3b883ed9d02a210e4dd27dc790539"
server_key_version: "kuduclusterkey@0"
I20260928 23:52:05.301806 19252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 23:52:05.328027 19268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:05.329713 19252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:52:05.329840 19252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "45f06518c2b54ae391c9ce91433fd025"
format_stamp: "Formatted at 2026-09-28 23:52:05 on dist-test-slave-p5bw"
server_key: "c4bfd5a936bf5d0510069dc0a2bfee0a"
server_key_iv: "bad3b883ed9d02a210e4dd27dc790539"
server_key_version: "kuduclusterkey@0"
I20260928 23:52:05.329977 19252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:05.356383 19252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:05.360144 19252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:05.360389 19252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:05.361135 19252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:52:05.362182 19252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:52:05.362274 19252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:05.362358 19252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:52:05.362409 19252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:05.372337 19252 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:36387
I20260928 23:52:05.373509 19252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:52:05.375890 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 19252
I20260928 23:52:05.376010 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:52:05.376322 6752 external_mini_cluster.cc:1428] Setting key ee95ff831c95772f3a2cb7ea8895c420
I20260928 23:52:05.387666 19381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:36387 every 8 connection(s)
WARNING: no policy specified for kudu/127.6.152.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.3@KRBTEST.COM" created.
Sep 28 23:52:05 dist-test-slave-p5bw krb5kdc[18555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639525, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:52:05.412041 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:05.392494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:60343 (local address 127.6.152.62:41649)
0928 23:52:05.392655 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:05.392661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:05.397456 (+ 4795us) server_negotiation.cc:407] Connection header received
0928 23:52:05.398786 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:05.398791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:05.398856 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:05.398917 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:05.400723 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:05.401579 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:05.403415 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:05.403771 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:05.403853 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:05.407138 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:05.407142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:05.407147 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:05.407180 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:05.409357 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:05.410032 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:05.410036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:05.410038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:05.410147 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:05.410597 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:05.410601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:05.410603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:05.410839 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:05.410915 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:05.411695 (+ 780us) server_negotiation.cc:299] Negotiation successful
0928 23:52:05.411840 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:52:05.413371 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:05.391388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:41649 (local address 127.6.152.2:60343)
0928 23:52:05.397249 (+ 5861us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:05.397323 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:05.398491 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:05.399398 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:05.399409 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:05.399794 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:05.400538 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:05.400559 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:05.401751 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:05.401754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:05.403266 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:05.403278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:05.404100 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:05.404105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:05.404317 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:05.405077 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:05.405109 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:05.406907 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:05.409502 (+ 2595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:05.409510 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:05.409530 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:05.409904 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:05.410273 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:05.410276 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:05.410278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:05.410478 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:05.411001 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:05.411008 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:05.411389 (+ 381us) client_negotiation.cc:769] Sending connection context
0928 23:52:05.412001 (+ 612us) client_negotiation.cc:240] Negotiation successful
0928 23:52:05.412461 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5459,"thread_start_us":149,"threads_started":1}
I20260928 23:52:05.414688 19382 heartbeater.cc:347] Connected to a master server at 127.6.152.62:41649
I20260928 23:52:05.414953 19382 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:05.415493 19382 heartbeater.cc:511] Master 127.6.152.62:41649 requested a full tablet report, sending...
I20260928 23:52:05.416590 19032 ts_manager.cc:194] Registered new tserver with Master: 45f06518c2b54ae391c9ce91433fd025 (127.6.152.2:36387)
I20260928 23:52:05.417348 19032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.2@KRBTEST.COM'} at 127.6.152.2:60343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
I20260928 23:52:05.428169 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.152.20:38615/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:41649
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:52:05.563287 19387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:05.563568 19387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:05.563651 19387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:05.567934 19387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:05.568037 19387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:05.568140 19387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:52:05.573673 19387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.152.20:38615/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:41649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:05.574826 19387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:05.575831 19387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:05.587174 19392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:05.587627 19393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:05.587920 19395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:05.588143 19387 server_base.cc:1034] running on GCE node
I20260928 23:52:05.588531 19387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:05.589313 19387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:05.590497 19387 hybrid_clock.cc:648] HybridClock initialized: now 1790639525590474 us; error 37 us; skew 500 ppm
Sep 28 23:52:05 dist-test-slave-p5bw krb5kdc[18555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639525, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:05.594049 19387 init.cc:377] Logged in from keytab as kudu/127.6.152.3@KRBTEST.COM (short username kudu)
I20260928 23:52:05.595155 19387 webserver.cc:460] Webserver started at http://127.6.152.3:44867/ using document root <none> and password file <none>
I20260928 23:52:05.595837 19387 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:05.595917 19387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:05.596145 19387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:52:05 dist-test-slave-p5bw krb5kdc[18555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639525, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for HTTP/127.6.152.20@KRBTEST.COM
I20260928 23:52:05.623387 19387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "9d63f63a2c8b498a814d8a81ea8d8180"
format_stamp: "Formatted at 2026-09-28 23:52:05 on dist-test-slave-p5bw"
server_key: "0bda5647b2f65da444902b9dabc693d7"
server_key_iv: "5123b1485538306e3216d626ec90d2cf"
server_key_version: "kuduclusterkey@0"
I20260928 23:52:05.624229 19387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "9d63f63a2c8b498a814d8a81ea8d8180"
format_stamp: "Formatted at 2026-09-28 23:52:05 on dist-test-slave-p5bw"
server_key: "0bda5647b2f65da444902b9dabc693d7"
server_key_iv: "5123b1485538306e3216d626ec90d2cf"
server_key_version: "kuduclusterkey@0"
I20260928 23:52:05.629307 19387 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
Sep 28 23:52:05 dist-test-slave-p5bw krb5kdc[18555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639525, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for HTTP/127.6.152.20@KRBTEST.COM
I20260928 23:52:05.668946 19402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:05.670867 19387 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.003s sys 0.000s
I20260928 23:52:05.671015 19387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "9d63f63a2c8b498a814d8a81ea8d8180"
format_stamp: "Formatted at 2026-09-28 23:52:05 on dist-test-slave-p5bw"
server_key: "0bda5647b2f65da444902b9dabc693d7"
server_key_iv: "5123b1485538306e3216d626ec90d2cf"
server_key_version: "kuduclusterkey@0"
I20260928 23:52:05.671177 19387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:05.746835 19387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:05.750497 19387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:05.750696 19387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:05.751412 19387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:52:05.752434 19387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:52:05.752498 19387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:05.752552 19387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:52:05.752647 19387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 23:52:05.762452 19387 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:46325
I20260928 23:52:05.762737 19515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:46325 every 8 connection(s)
I20260928 23:52:05.763629 19387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:52:05.766773 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 19387
I20260928 23:52:05.766899 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:52:05.767202 6752 external_mini_cluster.cc:1428] Setting key 21f07c6d98dc778e6eba01b781ecb9fd
Sep 28 23:52:05 dist-test-slave-p5bw krb5kdc[18555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639525, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:52:05.783655 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:05.771418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:39767 (local address 127.6.152.62:41649)
0928 23:52:05.771541 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:05.771546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:05.771561 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:52:05.772330 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:05.772333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:05.772383 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:05.772464 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:05.773836 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:05.774380 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:05.775545 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:05.775785 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:05.775900 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:05.779448 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:05.779452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:05.779455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:05.779483 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:05.781246 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:05.782016 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:05.782028 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:05.782030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:05.782128 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:05.782515 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:05.782520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:05.782522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:05.782720 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:05.782834 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:05.783359 (+ 525us) server_negotiation.cc:299] Negotiation successful
0928 23:52:05.783478 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:52:05.784565 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:05.770370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:41649 (local address 127.6.152.3:39767)
0928 23:52:05.770986 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:05.771053 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:05.772105 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:05.772688 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:05.772698 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:05.773093 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:05.773667 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:05.773682 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:05.774512 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:05.774516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:05.775406 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:05.775416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:05.775936 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:05.775941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:05.776197 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:05.776975 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:05.776991 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:05.779279 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:05.781522 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:05.781528 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:05.781544 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:05.781867 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:05.782241 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:05.782244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:05.782246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:05.782413 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:05.782833 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:05.782840 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:05.783114 (+ 274us) client_negotiation.cc:769] Sending connection context
0928 23:52:05.783317 (+ 203us) client_negotiation.cc:240] Negotiation successful
0928 23:52:05.783631 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":148,"threads_started":1}
I20260928 23:52:05.785746 19516 heartbeater.cc:347] Connected to a master server at 127.6.152.62:41649
I20260928 23:52:05.785985 19516 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:05.786475 19516 heartbeater.cc:511] Master 127.6.152.62:41649 requested a full tablet report, sending...
I20260928 23:52:05.787537 19032 ts_manager.cc:194] Registered new tserver with Master: 9d63f63a2c8b498a814d8a81ea8d8180 (127.6.152.3:46325)
I20260928 23:52:05.788270 19032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.3@KRBTEST.COM'} at 127.6.152.3:39767
I20260928 23:52:05.793227 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:52:05.794030 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 19113
I20260928 23:52:05.800138 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 19252
I20260928 23:52:05.805861 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 19387
I20260928 23:52:05.812863 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 19000
I20260928 23:52:05.818209 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:41649
--webserver_interface=127.6.152.62
--webserver_port=33559
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:41649
--ranger_config_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.152.20:38615/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.152.20:38615/kms with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:52:05.929409 19519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:05.929668 19519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:05.929706 19519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:05.933634 19519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:52:05.933688 19519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:05.933707 19519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:05.933723 19519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:52:05.933738 19519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:52:05.938998 19519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.152.20:38615/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:41649
--ranger_config_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:41649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=33559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:05.939939 19519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:05.940894 19519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:05.949108 19524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:05.949126 19525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:05.949189 19527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:05.949523 19519 server_base.cc:1034] running on GCE node
I20260928 23:52:05.950042 19519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:05.951097 19519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:05.952320 19519 hybrid_clock.cc:648] HybridClock initialized: now 1790639525952306 us; error 26 us; skew 500 ppm
Sep 28 23:52:05 dist-test-slave-p5bw krb5kdc[18555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639525, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:05.955122 19519 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:52:05.955961 19519 webserver.cc:460] Webserver started at http://127.6.152.62:33559/ using document root <none> and password file <none>
I20260928 23:52:05.956497 19519 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:05.956594 19519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 23:52:05 dist-test-slave-p5bw krb5kdc[18555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639525, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for HTTP/127.6.152.20@KRBTEST.COM
I20260928 23:52:06.008785 19519 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 23:52:06.011323 19535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:06.012600 19519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 23:52:06.012751 19519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "2e2100011350430789104bff84487749"
format_stamp: "Formatted at 2026-09-28 23:52:03 on dist-test-slave-p5bw"
server_key: "8a7eefb34215daf06927fa74e3a0f676"
server_key_iv: "1dcbe52b642baf75593acb364b2a51ec"
server_key_version: "kuduclusterkey@0"
I20260928 23:52:06.013363 19519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:06.026855 19519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:06.029735 19519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:06.029942 19519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:06.037165 19519 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:41649
I20260928 23:52:06.037199 19587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:41649 every 8 connection(s)
I20260928 23:52:06.038239 19519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:52:06.042520 19588 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:52:06.044144 19588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749: Bootstrap starting.
I20260928 23:52:06.045312 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 19519
I20260928 23:52:06.045859 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.152.20:38615/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:40751
--local_ip_for_outbound_sockets=127.6.152.1
--tserver_master_addrs=127.6.152.62:41649
--webserver_port=36709
--webserver_interface=127.6.152.1
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.152.20:38615/kms with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:52:06.060555 19588 log.cc:824] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:06.067137 19588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:52:06.067544 19588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749: Bootstrap complete.
I20260928 23:52:06.071651 19588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2100011350430789104bff84487749" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41649 } }
I20260928 23:52:06.072054 19588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2100011350430789104bff84487749, State: Initialized, Role: FOLLOWER
I20260928 23:52:06.072577 19588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2100011350430789104bff84487749" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41649 } }
I20260928 23:52:06.072724 19588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:52:06.072813 19588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:52:06.072958 19588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:52:06.074813 19588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2100011350430789104bff84487749" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41649 } }
I20260928 23:52:06.075153 19588 leader_election.cc:302] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e2100011350430789104bff84487749; no voters:
I20260928 23:52:06.075489 19588 leader_election.cc:288] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 23:52:06.075577 19592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 23:52:06.075834 19592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [term 2 LEADER]: Becoming Leader. State: Replica: 2e2100011350430789104bff84487749, State: Running, Role: LEADER
I20260928 23:52:06.076190 19592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2100011350430789104bff84487749" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41649 } }
I20260928 23:52:06.076704 19588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:52:06.077512 19593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2e2100011350430789104bff84487749" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2100011350430789104bff84487749" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41649 } } }
I20260928 23:52:06.077646 19593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [sys.catalog]: This master's current role is: LEADER
I20260928 23:52:06.078331 19594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e2100011350430789104bff84487749. Latest consensus state: current_term: 2 leader_uuid: "2e2100011350430789104bff84487749" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2100011350430789104bff84487749" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41649 } } }
I20260928 23:52:06.078460 19594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749 [sys.catalog]: This master's current role is: LEADER
I20260928 23:52:06.078862 19600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:52:06.079124 19588 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 23:52:06.085044 19600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:52:06.085768 19600 catalog_manager.cc:1259] Loaded cluster ID: 15dfd72b1a734f4f9eb99c92e4af5ab1
I20260928 23:52:06.085824 19600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:52:06.088655 19600 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:52:06.089396 19600 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749: Loaded TSK: 0
I20260928 23:52:06.090914 19600 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:52:06.187070 19590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:06.187495 19590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:06.187614 19590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:06.193653 19590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:06.193732 19590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:06.193821 19590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:52:06.201288 19590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.152.20:38615/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:40751
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=36709
--tserver_master_addrs=127.6.152.62:41649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:06.202690 19590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:06.203940 19590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:06.216357 19625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:06.216586 19626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:06.216094 19628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:06.216961 19590 server_base.cc:1034] running on GCE node
I20260928 23:52:06.217397 19590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:06.218175 19590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:06.219360 19590 hybrid_clock.cc:648] HybridClock initialized: now 1790639526219343 us; error 33 us; skew 500 ppm
Sep 28 23:52:06 dist-test-slave-p5bw krb5kdc[18555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639526, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:06.223018 19590 init.cc:377] Logged in from keytab as kudu/127.6.152.1@KRBTEST.COM (short username kudu)
I20260928 23:52:06.224114 19590 webserver.cc:460] Webserver started at http://127.6.152.1:36709/ using document root <none> and password file <none>
I20260928 23:52:06.224853 19590 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:06.225005 19590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 23:52:06 dist-test-slave-p5bw krb5kdc[18555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639526, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for HTTP/127.6.152.20@KRBTEST.COM
I20260928 23:52:06.254621 19590 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 23:52:06.257881 19636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:06.259454 19590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:52:06.259577 19590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "cea9be83626d45f88737ca8c35e5ef5d"
format_stamp: "Formatted at 2026-09-28 23:52:04 on dist-test-slave-p5bw"
server_key: "1d52ec25b0a843f72f0f22cc12af222d"
server_key_iv: "45261ef3959b9c4e25aa2ac7f33b27b0"
server_key_version: "kuduclusterkey@0"
I20260928 23:52:06.260269 19590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:06.283283 19590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:06.286952 19590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:06.287209 19590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:06.287936 19590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:52:06.289124 19590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:52:06.289242 19590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:06.289367 19590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:52:06.289449 19590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:06.300544 19590 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:40751
I20260928 23:52:06.301503 19749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:40751 every 8 connection(s)
I20260928 23:52:06.301769 19590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:52:06.305939 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 19590
I20260928 23:52:06.306358 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.152.20:38615/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:36387
--local_ip_for_outbound_sockets=127.6.152.2
--tserver_master_addrs=127.6.152.62:41649
--webserver_port=38947
--webserver_interface=127.6.152.2
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.152.20:38615/kms with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:52:06 dist-test-slave-p5bw krb5kdc[18555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639526, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:52:06.328052 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:06.304198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:38745 (local address 127.6.152.62:41649)
0928 23:52:06.304741 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:06.304763 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:06.309162 (+ 4399us) server_negotiation.cc:407] Connection header received
0928 23:52:06.310970 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:06.310990 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:06.311466 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:06.311824 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:06.314227 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:06.315963 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:06.318191 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:06.318903 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:06.319028 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:06.322006 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:06.322013 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:06.322029 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:06.322061 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:06.324172 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:06.324792 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:06.324798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:06.324804 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:06.324962 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:06.325342 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:06.325345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:06.325347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:06.325797 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:06.325921 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:06.326722 (+ 801us) server_negotiation.cc:299] Negotiation successful
0928 23:52:06.327034 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":111,"threads_started":1}
I20260928 23:52:06.330006 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:06.304210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:41649 (local address 127.6.152.1:38745)
0928 23:52:06.308959 (+ 4749us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:06.309021 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:06.310396 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:06.312488 (+ 2092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:06.312500 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:06.312901 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:06.313626 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:06.313650 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:06.316150 (+ 2500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:06.316156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:06.318054 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:06.318067 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:06.319278 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:06.319281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:06.319456 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:06.320125 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:06.320146 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:06.321786 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:06.324316 (+ 2530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:06.324322 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:06.324337 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:06.324661 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:06.325070 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:06.325073 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:06.325075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:06.325237 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:06.325996 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:06.326004 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:06.326367 (+ 363us) client_negotiation.cc:769] Sending connection context
0928 23:52:06.328801 (+ 2434us) client_negotiation.cc:240] Negotiation successful
0928 23:52:06.329132 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4481,"thread_start_us":115,"threads_started":1}
I20260928 23:52:06.332864 19750 heartbeater.cc:347] Connected to a master server at 127.6.152.62:41649
I20260928 23:52:06.333042 19750 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:06.333600 19750 heartbeater.cc:511] Master 127.6.152.62:41649 requested a full tablet report, sending...
W20260928 23:52:06.335983 19750 heartbeater.cc:644] Failed to heartbeat to 127.6.152.62:41649 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 23:52:06.336090 19750 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:06.336261 19750 heartbeater.cc:511] Master 127.6.152.62:41649 requested a full tablet report, sending...
I20260928 23:52:06.337247 19750 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:06.337414 19750 heartbeater.cc:511] Master 127.6.152.62:41649 requested a full tablet report, sending...
W20260928 23:52:06.338294 19750 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:52:06.441193 19754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:06.441474 19754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:06.441574 19754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:06.445720 19754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:06.445797 19754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:06.445881 19754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:52:06.451108 19754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.152.20:38615/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:36387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=38947
--tserver_master_addrs=127.6.152.62:41649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:06.452135 19754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:06.453365 19754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:06.463034 19760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:06.463037 19759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:06.463188 19762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:06.463419 19754 server_base.cc:1034] running on GCE node
I20260928 23:52:06.463846 19754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:06.464526 19754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:06.465710 19754 hybrid_clock.cc:648] HybridClock initialized: now 1790639526465697 us; error 29 us; skew 500 ppm
Sep 28 23:52:06 dist-test-slave-p5bw krb5kdc[18555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639526, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:06.468415 19754 init.cc:377] Logged in from keytab as kudu/127.6.152.2@KRBTEST.COM (short username kudu)
I20260928 23:52:06.469348 19754 webserver.cc:460] Webserver started at http://127.6.152.2:38947/ using document root <none> and password file <none>
I20260928 23:52:06.469888 19754 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:06.470031 19754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 23:52:06 dist-test-slave-p5bw krb5kdc[18555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639526, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for HTTP/127.6.152.20@KRBTEST.COM
I20260928 23:52:06.496721 19754 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 23:52:06.500396 19770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:06.502100 19754 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 23:52:06.502298 19754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "45f06518c2b54ae391c9ce91433fd025"
format_stamp: "Formatted at 2026-09-28 23:52:05 on dist-test-slave-p5bw"
server_key: "c4bfd5a936bf5d0510069dc0a2bfee0a"
server_key_iv: "bad3b883ed9d02a210e4dd27dc790539"
server_key_version: "kuduclusterkey@0"
I20260928 23:52:06.503121 19754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:06.536913 19754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:06.540575 19754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:06.540876 19754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:06.541675 19754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:52:06.542775 19754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:52:06.542845 19754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:06.542904 19754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:52:06.542939 19754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:06.554438 19754 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:36387
I20260928 23:52:06.554513 19883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:36387 every 8 connection(s)
I20260928 23:52:06.555830 19754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:52:06.557367 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 19754
I20260928 23:52:06.557853 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.152.20:38615/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:46325
--local_ip_for_outbound_sockets=127.6.152.3
--tserver_master_addrs=127.6.152.62:41649
--webserver_port=44867
--webserver_interface=127.6.152.3
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.152.20:38615/kms with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:52:06 dist-test-slave-p5bw krb5kdc[18555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639526, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:52:06.591015 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:06.558666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:47287 (local address 127.6.152.62:41649)
0928 23:52:06.558800 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:06.558804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:06.570976 (+ 12172us) server_negotiation.cc:407] Connection header received
0928 23:52:06.572538 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:06.572542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:06.572610 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:06.572727 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:06.574282 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:06.575153 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:06.576930 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:06.577298 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:06.577386 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:06.586722 (+ 9336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:06.586726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:06.586729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:06.586761 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:06.588534 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:06.589213 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:06.589217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:06.589219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:06.589320 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:06.589710 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:06.589714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:06.589716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:06.589946 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:06.590028 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:06.590691 (+ 663us) server_negotiation.cc:299] Negotiation successful
0928 23:52:06.590838 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:52:06.591775 19888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:06.559126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:41649 (local address 127.6.152.2:47287)
0928 23:52:06.570711 (+ 11585us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:06.570798 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:06.572223 (+ 1425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:06.573016 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:06.573026 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:06.573411 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:06.574108 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:06.574125 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:06.575323 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:06.575327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:06.576766 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:06.576778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:06.577607 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:06.577613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:06.577824 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:06.582787 (+ 4963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:06.582805 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:06.586497 (+ 3692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:06.588698 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:06.588703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:06.588718 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:06.589060 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:06.589429 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:06.589432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:06.589434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:06.589594 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:06.590092 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:06.590098 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:06.590397 (+ 299us) client_negotiation.cc:769] Sending connection context
0928 23:52:06.590582 (+ 185us) client_negotiation.cc:240] Negotiation successful
0928 23:52:06.590881 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11255,"thread_start_us":146,"threads_started":1}
I20260928 23:52:06.593096 19885 heartbeater.cc:347] Connected to a master server at 127.6.152.62:41649
I20260928 23:52:06.593353 19885 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:06.593902 19885 heartbeater.cc:511] Master 127.6.152.62:41649 requested a full tablet report, sending...
W20260928 23:52:06.595577 19885 heartbeater.cc:644] Failed to heartbeat to 127.6.152.62:41649 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 23:52:06.595685 19885 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:06.595832 19885 heartbeater.cc:511] Master 127.6.152.62:41649 requested a full tablet report, sending...
I20260928 23:52:06.596602 19885 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:06.596751 19885 heartbeater.cc:511] Master 127.6.152.62:41649 requested a full tablet report, sending...
W20260928 23:52:06.597491 19885 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:52:06.701891 19887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:06.702179 19887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:06.702287 19887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:06.707377 19887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:06.707474 19887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:06.707576 19887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:52:06.713250 19887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.152.20:38615/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:46325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=44867
--tserver_master_addrs=127.6.152.62:41649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:06.714380 19887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:06.715456 19887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:06.727203 19894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:06.727203 19893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:06.728381 19896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:06.729009 19887 server_base.cc:1034] running on GCE node
I20260928 23:52:06.729439 19887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:06.730080 19887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:06.731273 19887 hybrid_clock.cc:648] HybridClock initialized: now 1790639526731254 us; error 44 us; skew 500 ppm
Sep 28 23:52:06 dist-test-slave-p5bw krb5kdc[18555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639526, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:06.734395 19887 init.cc:377] Logged in from keytab as kudu/127.6.152.3@KRBTEST.COM (short username kudu)
I20260928 23:52:06.735548 19887 webserver.cc:460] Webserver started at http://127.6.152.3:44867/ using document root <none> and password file <none>
I20260928 23:52:06.736181 19887 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:06.736299 19887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 23:52:06 dist-test-slave-p5bw krb5kdc[18555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639526, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for HTTP/127.6.152.20@KRBTEST.COM
I20260928 23:52:06.755882 19887 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260928 23:52:06.759778 19904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:06.761381 19887 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 23:52:06.761529 19887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "9d63f63a2c8b498a814d8a81ea8d8180"
format_stamp: "Formatted at 2026-09-28 23:52:05 on dist-test-slave-p5bw"
server_key: "0bda5647b2f65da444902b9dabc693d7"
server_key_iv: "5123b1485538306e3216d626ec90d2cf"
server_key_version: "kuduclusterkey@0"
I20260928 23:52:06.762108 19887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:06.774194 19887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:06.778404 19887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:06.778652 19887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:06.779271 19887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:52:06.780228 19887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:52:06.780299 19887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:06.780393 19887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:52:06.780449 19887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:06.791989 19887 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:46325
I20260928 23:52:06.792022 20017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:46325 every 8 connection(s)
I20260928 23:52:06.793270 19887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:52:06.798115 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 19887
Sep 28 23:52:06 dist-test-slave-p5bw krb5kdc[18555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639526, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:52:06.810526 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:06.798764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49838 (local address 127.6.152.62:41649)
0928 23:52:06.798991 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:06.798996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:06.799258 (+ 262us) server_negotiation.cc:407] Connection header received
0928 23:52:06.799433 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:06.799436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:06.799477 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:06.799538 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:06.800532 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:06.801437 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:06.802672 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:06.803011 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:06.803096 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:06.805940 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:06.805947 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:06.805952 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:06.805981 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:06.808229 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:06.809013 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:06.809019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:06.809024 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:06.809114 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:06.809433 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:06.809438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:06.809440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:06.809664 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:06.809834 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:06.810216 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 23:52:06.810342 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":167,"thread_start_us":79,"threads_started":1}
I20260928 23:52:06.810904 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:06.797045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:33265 (local address 127.6.152.62:41649)
0928 23:52:06.797217 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:06.797221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:06.797405 (+ 184us) server_negotiation.cc:407] Connection header received
0928 23:52:06.798786 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:06.798789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:06.798844 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:06.798907 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:06.800607 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:06.801437 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:06.803031 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:06.803344 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:06.803413 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:06.806259 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:06.806264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:06.806269 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:06.806317 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:06.808476 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:06.809186 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:06.809192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:06.809196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:06.809270 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:06.809673 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:06.809676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:06.809678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:06.809845 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:06.809941 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:06.810652 (+ 711us) server_negotiation.cc:299] Negotiation successful
0928 23:52:06.810773 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":65}
I20260928 23:52:06.812410 20020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:06.795840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:41649 (local address 127.6.152.3:33265)
0928 23:52:06.797240 (+ 1400us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:06.797299 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:06.798457 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:06.799358 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:06.799370 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:06.799740 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:06.800465 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:06.800487 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:06.801587 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:06.801590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:06.802910 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:06.802920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:06.803604 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:06.803611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:06.803829 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:06.804473 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:06.804494 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:06.806052 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:06.808589 (+ 2537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:06.808596 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:06.808611 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:06.808984 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:06.809387 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:06.809393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:06.809397 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:06.809565 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:06.810032 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:06.810038 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:06.810376 (+ 338us) client_negotiation.cc:769] Sending connection context
0928 23:52:06.811014 (+ 638us) client_negotiation.cc:240] Negotiation successful
0928 23:52:06.811371 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1133,"thread_start_us":137,"threads_started":1}
I20260928 23:52:06.813759 20018 heartbeater.cc:347] Connected to a master server at 127.6.152.62:41649
I20260928 23:52:06.813993 20018 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:06.814596 20018 heartbeater.cc:511] Master 127.6.152.62:41649 requested a full tablet report, sending...
W20260928 23:52:06.815994 20018 heartbeater.cc:644] Failed to heartbeat to 127.6.152.62:41649 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 23:52:06.816138 20018 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:06.816294 20018 heartbeater.cc:511] Master 127.6.152.62:41649 requested a full tablet report, sending...
I20260928 23:52:06.817164 20018 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:06.817340 20018 heartbeater.cc:511] Master 127.6.152.62:41649 requested a full tablet report, sending...
W20260928 23:52:06.818212 20018 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 23:52:07 dist-test-slave-p5bw krb5kdc[18555](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790639527, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:07.338534 19750 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:07.338862 19750 heartbeater.cc:511] Master 127.6.152.62:41649 requested a full tablet report, sending...
I20260928 23:52:07.339975 19552 ts_manager.cc:194] Registered new tserver with Master: cea9be83626d45f88737ca8c35e5ef5d (127.6.152.1:40751)
I20260928 23:52:07.341379 19552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.1@KRBTEST.COM'} at 127.6.152.1:38745
Sep 28 23:52:07 dist-test-slave-p5bw krb5kdc[18555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639527, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for HTTP/127.6.152.20@KRBTEST.COM
I20260928 23:52:07.597704 19885 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:07.598021 19885 heartbeater.cc:511] Master 127.6.152.62:41649 requested a full tablet report, sending...
I20260928 23:52:07.598802 19552 ts_manager.cc:194] Registered new tserver with Master: 45f06518c2b54ae391c9ce91433fd025 (127.6.152.2:36387)
I20260928 23:52:07.599606 19552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.2@KRBTEST.COM'} at 127.6.152.2:47287
I20260928 23:52:07.818408 20018 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:07.818735 20018 heartbeater.cc:511] Master 127.6.152.62:41649 requested a full tablet report, sending...
I20260928 23:52:07.819410 19552 ts_manager.cc:194] Registered new tserver with Master: 9d63f63a2c8b498a814d8a81ea8d8180 (127.6.152.3:46325)
I20260928 23:52:07.820119 19552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.3@KRBTEST.COM'} at 127.6.152.3:33265
I20260928 23:52:07.830325 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:52:07.844144 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:07.834854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49848 (local address 127.6.152.62:41649)
0928 23:52:07.835117 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:07.835122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:07.835136 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:52:07.835173 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:07.835176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:07.835223 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:07.836093 (+ 870us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:07.836188 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:07.837002 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:07.838138 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:07.838499 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:07.840265 (+ 1766us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:07.840381 (+ 116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:07.840384 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:07.840387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:07.840404 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:07.842390 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:07.842892 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:07.842895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:07.842898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:07.842980 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:07.843278 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:07.843283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:07.843285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:07.843456 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:07.843590 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:07.843835 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 23:52:07.843954 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":89,"threads_started":1}
I20260928 23:52:07.848425 19552 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
Sep 28 23:52:07 dist-test-slave-p5bw krb5kdc[18555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639527, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for HTTP/127.6.152.20@KRBTEST.COM
W20260928 23:52:07.852829 19552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:52:07.892599 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:07.882146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:36387 (local address 127.0.0.1:51536)
0928 23:52:07.884154 (+ 2008us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:07.884210 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:07.884335 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:07.886285 (+ 1950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:07.886288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:07.886307 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:52:07.886445 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:07.886456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:07.888671 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:07.888675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:07.890028 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:07.890039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:07.892048 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:07.892052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:07.892164 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:07.892205 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 23:52:07.892316 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 23:52:07.892433 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1807,"mutex_wait_us":21,"thread_start_us":104,"threads_started":1}
I20260928 23:52:07.893028 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:07.882990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49138 (local address 127.6.152.3:46325)
0928 23:52:07.884894 (+ 1904us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:07.884901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:07.884927 (+ 26us) server_negotiation.cc:407] Connection header received
0928 23:52:07.885012 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:07.885018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:07.885245 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:07.885417 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:52:07.887049 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:07.888297 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:07.890920 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:07.892090 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:07.892182 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:07.892317 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:07.892726 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 23:52:07.892851 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1770,"thread_start_us":118,"threads_started":1}
I20260928 23:52:07.893468 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:07.883349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51536 (local address 127.6.152.2:36387)
0928 23:52:07.885733 (+ 2384us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:07.885740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:07.885761 (+ 21us) server_negotiation.cc:407] Connection header received
0928 23:52:07.885830 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:07.885835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:07.886003 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:07.886134 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:52:07.886582 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:07.888520 (+ 1938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:07.890176 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:07.890988 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:07.891086 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:07.891233 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:07.893230 (+ 1997us) server_negotiation.cc:299] Negotiation successful
0928 23:52:07.893328 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2268,"thread_start_us":86,"threads_started":1}
I20260928 23:52:07.894420 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:07.882269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:46325 (local address 127.0.0.1:49138)
0928 23:52:07.884551 (+ 2282us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:07.884563 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:07.884630 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:07.885550 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:07.885557 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:07.885583 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:52:07.886386 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:07.886396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:07.888585 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:07.888588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:07.890774 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:07.890785 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:07.892374 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:07.892378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:07.892462 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:07.892482 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 23:52:07.892545 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 23:52:07.892606 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2224,"mutex_wait_us":1,"thread_start_us":111,"threads_started":1}
I20260928 23:52:07.896749 19818 tablet_service.cc:1467] Processing CreateTablet for tablet f3f3437a44c448bdb19e255aa9761768 (DEFAULT_TABLE table=test-table [id=c7fbb3603de74bf1a3f41be16834717e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:52:07.897416 19952 tablet_service.cc:1467] Processing CreateTablet for tablet f3f3437a44c448bdb19e255aa9761768 (DEFAULT_TABLE table=test-table [id=c7fbb3603de74bf1a3f41be16834717e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:52:07.897925 19818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3f3437a44c448bdb19e255aa9761768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:52:07.898367 19952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3f3437a44c448bdb19e255aa9761768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:52:07.907245 20053 tablet_bootstrap.cc:492] T f3f3437a44c448bdb19e255aa9761768 P 9d63f63a2c8b498a814d8a81ea8d8180: Bootstrap starting.
I20260928 23:52:07.908838 20054 tablet_bootstrap.cc:492] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025: Bootstrap starting.
I20260928 23:52:07.910622 20053 tablet_bootstrap.cc:654] T f3f3437a44c448bdb19e255aa9761768 P 9d63f63a2c8b498a814d8a81ea8d8180: Neither blocks nor log segments found. Creating new log.
I20260928 23:52:07.911518 20053 log.cc:824] T f3f3437a44c448bdb19e255aa9761768 P 9d63f63a2c8b498a814d8a81ea8d8180: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:07.915500 20053 tablet_bootstrap.cc:492] T f3f3437a44c448bdb19e255aa9761768 P 9d63f63a2c8b498a814d8a81ea8d8180: No bootstrap required, opened a new log
I20260928 23:52:07.915699 20053 ts_tablet_manager.cc:1397] T f3f3437a44c448bdb19e255aa9761768 P 9d63f63a2c8b498a814d8a81ea8d8180: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260928 23:52:07.918211 20054 tablet_bootstrap.cc:654] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025: Neither blocks nor log segments found. Creating new log.
I20260928 23:52:07.919013 20054 log.cc:824] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:07.919996 20053 raft_consensus.cc:348] T f3f3437a44c448bdb19e255aa9761768 P 9d63f63a2c8b498a814d8a81ea8d8180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f06518c2b54ae391c9ce91433fd025" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 36387 } } peers { permanent_uuid: "9d63f63a2c8b498a814d8a81ea8d8180" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46325 } } peers { permanent_uuid: "cea9be83626d45f88737ca8c35e5ef5d" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 40751 } }
I20260928 23:52:07.920234 20053 raft_consensus.cc:374] T f3f3437a44c448bdb19e255aa9761768 P 9d63f63a2c8b498a814d8a81ea8d8180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:52:07.920275 20053 raft_consensus.cc:729] T f3f3437a44c448bdb19e255aa9761768 P 9d63f63a2c8b498a814d8a81ea8d8180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d63f63a2c8b498a814d8a81ea8d8180, State: Initialized, Role: FOLLOWER
I20260928 23:52:07.922338 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:07.884039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:40751 (local address 127.0.0.1:54558)
0928 23:52:07.888864 (+ 4825us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:07.888878 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:07.888958 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:07.916013 (+ 27055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:07.916019 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:07.916052 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:52:07.916185 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:07.916194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:07.919462 (+ 3268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:07.919466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:07.920604 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:07.920615 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:07.921894 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:07.921898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:07.921998 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:07.922029 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 23:52:07.922084 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 23:52:07.922170 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4742,"thread_start_us":81,"threads_started":1}
I20260928 23:52:07.923316 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:07.884399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54558 (local address 127.6.152.1:40751)
0928 23:52:07.889198 (+ 4799us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:07.889205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:07.889623 (+ 418us) server_negotiation.cc:407] Connection header received
0928 23:52:07.889705 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:07.889709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:07.914769 (+ 25060us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:07.914939 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:52:07.916344 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:07.917608 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:07.920827 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:07.921754 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:07.922790 (+ 1036us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:07.922934 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:07.922989 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 23:52:07.923181 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4687,"thread_start_us":84,"threads_started":1}
I20260928 23:52:07.920710 20053 consensus_queue.cc:260] T f3f3437a44c448bdb19e255aa9761768 P 9d63f63a2c8b498a814d8a81ea8d8180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f06518c2b54ae391c9ce91433fd025" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 36387 } } peers { permanent_uuid: "9d63f63a2c8b498a814d8a81ea8d8180" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46325 } } peers { permanent_uuid: "cea9be83626d45f88737ca8c35e5ef5d" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 40751 } }
I20260928 23:52:07.925381 20053 ts_tablet_manager.cc:1428] T f3f3437a44c448bdb19e255aa9761768 P 9d63f63a2c8b498a814d8a81ea8d8180: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260928 23:52:07.925731 20018 heartbeater.cc:503] Master 127.6.152.62:41649 was elected leader, sending a full tablet report...
I20260928 23:52:07.926494 19684 tablet_service.cc:1467] Processing CreateTablet for tablet f3f3437a44c448bdb19e255aa9761768 (DEFAULT_TABLE table=test-table [id=c7fbb3603de74bf1a3f41be16834717e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:52:07.927573 19684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3f3437a44c448bdb19e255aa9761768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:52:07.937770 20054 tablet_bootstrap.cc:492] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025: No bootstrap required, opened a new log
I20260928 23:52:07.938040 20054 ts_tablet_manager.cc:1397] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.000s
I20260928 23:52:07.942646 20054 raft_consensus.cc:348] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f06518c2b54ae391c9ce91433fd025" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 36387 } } peers { permanent_uuid: "9d63f63a2c8b498a814d8a81ea8d8180" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46325 } } peers { permanent_uuid: "cea9be83626d45f88737ca8c35e5ef5d" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 40751 } }
I20260928 23:52:07.943145 20054 raft_consensus.cc:374] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:52:07.943078 20058 tablet_bootstrap.cc:492] T f3f3437a44c448bdb19e255aa9761768 P cea9be83626d45f88737ca8c35e5ef5d: Bootstrap starting.
I20260928 23:52:07.943395 20054 raft_consensus.cc:729] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f06518c2b54ae391c9ce91433fd025, State: Initialized, Role: FOLLOWER
I20260928 23:52:07.943997 20054 consensus_queue.cc:260] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f06518c2b54ae391c9ce91433fd025" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 36387 } } peers { permanent_uuid: "9d63f63a2c8b498a814d8a81ea8d8180" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46325 } } peers { permanent_uuid: "cea9be83626d45f88737ca8c35e5ef5d" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 40751 } }
I20260928 23:52:07.945184 20054 ts_tablet_manager.cc:1428] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 23:52:07.945591 19885 heartbeater.cc:503] Master 127.6.152.62:41649 was elected leader, sending a full tablet report...
I20260928 23:52:07.945962 20058 tablet_bootstrap.cc:654] T f3f3437a44c448bdb19e255aa9761768 P cea9be83626d45f88737ca8c35e5ef5d: Neither blocks nor log segments found. Creating new log.
I20260928 23:52:07.946910 20058 log.cc:824] T f3f3437a44c448bdb19e255aa9761768 P cea9be83626d45f88737ca8c35e5ef5d: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:07.949486 20058 tablet_bootstrap.cc:492] T f3f3437a44c448bdb19e255aa9761768 P cea9be83626d45f88737ca8c35e5ef5d: No bootstrap required, opened a new log
I20260928 23:52:07.949743 20058 ts_tablet_manager.cc:1397] T f3f3437a44c448bdb19e255aa9761768 P cea9be83626d45f88737ca8c35e5ef5d: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 23:52:07.953150 20058 raft_consensus.cc:348] T f3f3437a44c448bdb19e255aa9761768 P cea9be83626d45f88737ca8c35e5ef5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f06518c2b54ae391c9ce91433fd025" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 36387 } } peers { permanent_uuid: "9d63f63a2c8b498a814d8a81ea8d8180" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46325 } } peers { permanent_uuid: "cea9be83626d45f88737ca8c35e5ef5d" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 40751 } }
I20260928 23:52:07.953408 20058 raft_consensus.cc:374] T f3f3437a44c448bdb19e255aa9761768 P cea9be83626d45f88737ca8c35e5ef5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:52:07.953634 20058 raft_consensus.cc:729] T f3f3437a44c448bdb19e255aa9761768 P cea9be83626d45f88737ca8c35e5ef5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea9be83626d45f88737ca8c35e5ef5d, State: Initialized, Role: FOLLOWER
I20260928 23:52:07.954221 20058 consensus_queue.cc:260] T f3f3437a44c448bdb19e255aa9761768 P cea9be83626d45f88737ca8c35e5ef5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f06518c2b54ae391c9ce91433fd025" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 36387 } } peers { permanent_uuid: "9d63f63a2c8b498a814d8a81ea8d8180" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46325 } } peers { permanent_uuid: "cea9be83626d45f88737ca8c35e5ef5d" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 40751 } }
I20260928 23:52:07.954910 19750 heartbeater.cc:503] Master 127.6.152.62:41649 was elected leader, sending a full tablet report...
I20260928 23:52:07.955340 20058 ts_tablet_manager.cc:1428] T f3f3437a44c448bdb19e255aa9761768 P cea9be83626d45f88737ca8c35e5ef5d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:52:08.028539 20059 raft_consensus.cc:482] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:52:08.028800 20059 raft_consensus.cc:504] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f06518c2b54ae391c9ce91433fd025" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 36387 } } peers { permanent_uuid: "9d63f63a2c8b498a814d8a81ea8d8180" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46325 } } peers { permanent_uuid: "cea9be83626d45f88737ca8c35e5ef5d" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 40751 } }
I20260928 23:52:08.030388 20059 leader_election.cc:288] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d63f63a2c8b498a814d8a81ea8d8180 (127.6.152.3:46325), cea9be83626d45f88737ca8c35e5ef5d (127.6.152.1:40751)
W20260928 23:52:08.045380 20019 tablet.cc:2369] T f3f3437a44c448bdb19e255aa9761768 P 9d63f63a2c8b498a814d8a81ea8d8180: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:52:08.051338 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:08.031521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:43639 (local address 127.6.152.1:40751)
0928 23:52:08.031665 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:08.031671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:08.035710 (+ 4039us) server_negotiation.cc:407] Connection header received
0928 23:52:08.038755 (+ 3045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:08.038759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:08.038815 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:08.038863 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:52:08.041523 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:08.042198 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:08.049807 (+ 7609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:08.050364 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:08.050447 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:08.050503 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:08.051108 (+ 605us) server_negotiation.cc:299] Negotiation successful
0928 23:52:08.051204 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:52:08.052052 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:08.031357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:40751 (local address 127.6.152.2:43639)
0928 23:52:08.035628 (+ 4271us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:08.035640 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:08.038628 (+ 2988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:08.038988 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:08.038991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:08.039006 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:52:08.039123 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:08.039129 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:08.042338 (+ 3209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:08.042341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:08.043529 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:08.043540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:08.050883 (+ 7343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:08.050886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:08.050970 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:08.050989 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 23:52:08.051799 (+ 810us) client_negotiation.cc:240] Negotiation successful
0928 23:52:08.051886 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4206,"thread_start_us":89,"threads_started":1}
I20260928 23:52:08.052543 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:08.031736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:60635 (local address 127.6.152.3:46325)
0928 23:52:08.031844 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:08.031849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:08.035452 (+ 3603us) server_negotiation.cc:407] Connection header received
0928 23:52:08.037075 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:08.037079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:08.037132 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:08.037180 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:52:08.039253 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:08.040012 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:08.048903 (+ 8891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:08.049540 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:08.049643 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:08.049717 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:08.052332 (+ 2615us) server_negotiation.cc:299] Negotiation successful
0928 23:52:08.052427 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 23:52:08.053108 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:08.030939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:46325 (local address 127.6.152.2:60635)
0928 23:52:08.035341 (+ 4402us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:08.035357 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:08.036915 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:08.038332 (+ 1417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:08.038335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:08.038354 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:52:08.038498 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:08.038504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:08.040152 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:08.040154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:08.041367 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:08.041379 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:08.050619 (+ 9240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:08.050622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:08.050736 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:08.050759 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:52:08.052897 (+ 2138us) client_negotiation.cc:240] Negotiation successful
0928 23:52:08.052976 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4320,"thread_start_us":123,"threads_started":1}
I20260928 23:52:08.053838 19704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f3437a44c448bdb19e255aa9761768" candidate_uuid: "45f06518c2b54ae391c9ce91433fd025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea9be83626d45f88737ca8c35e5ef5d" is_pre_election: true
I20260928 23:52:08.054203 19704 raft_consensus.cc:2454] T f3f3437a44c448bdb19e255aa9761768 P cea9be83626d45f88737ca8c35e5ef5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45f06518c2b54ae391c9ce91433fd025 in term 0.
I20260928 23:52:08.054229 19970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f3437a44c448bdb19e255aa9761768" candidate_uuid: "45f06518c2b54ae391c9ce91433fd025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d63f63a2c8b498a814d8a81ea8d8180" is_pre_election: true
I20260928 23:52:08.054498 19970 raft_consensus.cc:2454] T f3f3437a44c448bdb19e255aa9761768 P 9d63f63a2c8b498a814d8a81ea8d8180 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45f06518c2b54ae391c9ce91433fd025 in term 0.
I20260928 23:52:08.054770 19774 leader_election.cc:302] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45f06518c2b54ae391c9ce91433fd025, cea9be83626d45f88737ca8c35e5ef5d; no voters:
I20260928 23:52:08.055362 20059 raft_consensus.cc:2781] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:52:08.055446 20059 raft_consensus.cc:482] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:52:08.055491 20059 raft_consensus.cc:3037] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:52:08.056943 20059 raft_consensus.cc:504] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f06518c2b54ae391c9ce91433fd025" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 36387 } } peers { permanent_uuid: "9d63f63a2c8b498a814d8a81ea8d8180" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46325 } } peers { permanent_uuid: "cea9be83626d45f88737ca8c35e5ef5d" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 40751 } }
I20260928 23:52:08.057317 20059 leader_election.cc:288] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025 [CANDIDATE]: Term 1 election: Requested vote from peers 9d63f63a2c8b498a814d8a81ea8d8180 (127.6.152.3:46325), cea9be83626d45f88737ca8c35e5ef5d (127.6.152.1:40751)
I20260928 23:52:08.057904 19970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f3437a44c448bdb19e255aa9761768" candidate_uuid: "45f06518c2b54ae391c9ce91433fd025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d63f63a2c8b498a814d8a81ea8d8180"
I20260928 23:52:08.058027 19970 raft_consensus.cc:3037] T f3f3437a44c448bdb19e255aa9761768 P 9d63f63a2c8b498a814d8a81ea8d8180 [term 0 FOLLOWER]: Advancing to term 1
W20260928 23:52:08.058169 19886 tablet.cc:2369] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:52:08.059321 19970 raft_consensus.cc:2454] T f3f3437a44c448bdb19e255aa9761768 P 9d63f63a2c8b498a814d8a81ea8d8180 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45f06518c2b54ae391c9ce91433fd025 in term 1.
I20260928 23:52:08.059708 19774 leader_election.cc:302] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45f06518c2b54ae391c9ce91433fd025, 9d63f63a2c8b498a814d8a81ea8d8180; no voters:
I20260928 23:52:08.059755 19704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f3437a44c448bdb19e255aa9761768" candidate_uuid: "45f06518c2b54ae391c9ce91433fd025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea9be83626d45f88737ca8c35e5ef5d"
I20260928 23:52:08.059899 19704 raft_consensus.cc:3037] T f3f3437a44c448bdb19e255aa9761768 P cea9be83626d45f88737ca8c35e5ef5d [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:52:08.059958 20059 raft_consensus.cc:2781] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:52:08.060175 20059 raft_consensus.cc:686] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025 [term 1 LEADER]: Becoming Leader. State: Replica: 45f06518c2b54ae391c9ce91433fd025, State: Running, Role: LEADER
I20260928 23:52:08.060530 20059 consensus_queue.cc:237] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f06518c2b54ae391c9ce91433fd025" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 36387 } } peers { permanent_uuid: "9d63f63a2c8b498a814d8a81ea8d8180" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46325 } } peers { permanent_uuid: "cea9be83626d45f88737ca8c35e5ef5d" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 40751 } }
I20260928 23:52:08.061244 19704 raft_consensus.cc:2454] T f3f3437a44c448bdb19e255aa9761768 P cea9be83626d45f88737ca8c35e5ef5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45f06518c2b54ae391c9ce91433fd025 in term 1.
I20260928 23:52:08.064282 19551 catalog_manager.cc:5495] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45f06518c2b54ae391c9ce91433fd025 (127.6.152.2). New cstate: current_term: 1 leader_uuid: "45f06518c2b54ae391c9ce91433fd025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45f06518c2b54ae391c9ce91433fd025" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 36387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d63f63a2c8b498a814d8a81ea8d8180" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cea9be83626d45f88737ca8c35e5ef5d" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 40751 } health_report { overall_health: UNKNOWN } } }
W20260928 23:52:08.065376 19751 tablet.cc:2369] T f3f3437a44c448bdb19e255aa9761768 P cea9be83626d45f88737ca8c35e5ef5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:52:08.084702 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:08.080002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51538 (local address 127.6.152.2:36387)
0928 23:52:08.080132 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:08.080136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:08.080153 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:52:08.080275 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:08.080278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:08.080348 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:08.080459 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:52:08.080785 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:08.081642 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:08.082867 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:08.083227 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:08.083334 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:08.083696 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:52:08.084249 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:52:08.084372 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:08.084469 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 23:52:08.084529 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 23:52:08.092684 19704 raft_consensus.cc:1261] T f3f3437a44c448bdb19e255aa9761768 P cea9be83626d45f88737ca8c35e5ef5d [term 1 FOLLOWER]: Refusing update from remote peer 45f06518c2b54ae391c9ce91433fd025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:52:08.092772 19970 raft_consensus.cc:1261] T f3f3437a44c448bdb19e255aa9761768 P 9d63f63a2c8b498a814d8a81ea8d8180 [term 1 FOLLOWER]: Refusing update from remote peer 45f06518c2b54ae391c9ce91433fd025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:52:08.093443 20059 consensus_queue.cc:1035] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea9be83626d45f88737ca8c35e5ef5d" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 40751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:52:08.093629 20064 consensus_queue.cc:1035] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d63f63a2c8b498a814d8a81ea8d8180" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:52:08.103111 20067 mvcc.cc:205] Tried to move back new op lower bound from 7334459507060076544 to 7334459506941374464. Current Snapshot: MvccSnapshot[applied={T|T < 7334459507060076544}]
I20260928 23:52:08.106241 20069 mvcc.cc:205] Tried to move back new op lower bound from 7334459507060076544 to 7334459506941374464. Current Snapshot: MvccSnapshot[applied={T|T < 7334459507060076544}]
I20260928 23:52:08.114995 19552 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:52:08.115183 19552 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:52:08.118163 19552 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e2100011350430789104bff84487749: Sending DeleteTablet for 3 replicas of tablet f3f3437a44c448bdb19e255aa9761768
I20260928 23:52:08.119299 19684 tablet_service.cc:1514] Processing DeleteTablet for tablet f3f3437a44c448bdb19e255aa9761768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:52:08 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:54558
I20260928 23:52:08.119482 19818 tablet_service.cc:1514] Processing DeleteTablet for tablet f3f3437a44c448bdb19e255aa9761768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:52:08 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:51536
I20260928 23:52:08.120191 19952 tablet_service.cc:1514] Processing DeleteTablet for tablet f3f3437a44c448bdb19e255aa9761768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:52:08 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:49138
I20260928 23:52:08.121035 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 19590
I20260928 23:52:08.122680 20077 tablet_replica.cc:330] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025: stopping tablet replica
I20260928 23:52:08.122993 20077 raft_consensus.cc:2229] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:52:08.123350 20077 raft_consensus.cc:2258] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:52:08.127792 20077 ts_tablet_manager.cc:1905] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 23:52:08.128595 19539 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.152.1:40751 (error 108)
I20260928 23:52:08.130266 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:08.129936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:40751 (local address 127.0.0.1:54566)
0928 23:52:08.130113 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:08.130179 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.1:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260928 23:52:08.130558 19539 catalog_manager.cc:4584] TS cea9be83626d45f88737ca8c35e5ef5d (127.6.152.1:40751): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f3f3437a44c448bdb19e255aa9761768: Network error: Client connection negotiation failed: client connection to 127.6.152.1:40751: connect: Connection refused (error 111)
I20260928 23:52:08.130718 20077 ts_tablet_manager.cc:1918] T f3f3437a44c448bdb19e255aa9761768 P 45f06518c2b54ae391c9ce91433fd025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:52:08.130957 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 19754
I20260928 23:52:08.130983 20078 tablet_replica.cc:330] T f3f3437a44c448bdb19e255aa9761768 P 9d63f63a2c8b498a814d8a81ea8d8180: stopping tablet replica
I20260928 23:52:08.131305 20078 raft_consensus.cc:2229] T f3f3437a44c448bdb19e255aa9761768 P 9d63f63a2c8b498a814d8a81ea8d8180 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:52:08.131704 20078 raft_consensus.cc:2258] T f3f3437a44c448bdb19e255aa9761768 P 9d63f63a2c8b498a814d8a81ea8d8180 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:52:08.133399 20078 ts_tablet_manager.cc:1905] T f3f3437a44c448bdb19e255aa9761768 P 9d63f63a2c8b498a814d8a81ea8d8180: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:52:08.136572 20078 ts_tablet_manager.cc:1918] T f3f3437a44c448bdb19e255aa9761768 P 9d63f63a2c8b498a814d8a81ea8d8180: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:52:08.136731 20078 log.cc:1196] T f3f3437a44c448bdb19e255aa9761768 P 9d63f63a2c8b498a814d8a81ea8d8180: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/f3f3437a44c448bdb19e255aa9761768
I20260928 23:52:08.137180 20078 ts_tablet_manager.cc:1939] T f3f3437a44c448bdb19e255aa9761768 P 9d63f63a2c8b498a814d8a81ea8d8180: Deleting consensus metadata
I20260928 23:52:08.138751 19539 catalog_manager.cc:4841] TS 9d63f63a2c8b498a814d8a81ea8d8180 (127.6.152.3:46325): tablet f3f3437a44c448bdb19e255aa9761768 (table test-table [id=c7fbb3603de74bf1a3f41be16834717e]) successfully deleted
I20260928 23:52:08.141258 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 19887
I20260928 23:52:08.142515 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:08.142203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:36387 (local address 127.0.0.1:51540)
0928 23:52:08.142372 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:08.142426 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.2:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260928 23:52:08.147866 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 19519
I20260928 23:52:08.153712 6752 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 23:52:08.498303 6752 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 23:52:08.498406 6752 mini_ranger.cc:66] Stopping Ranger...
I20260928 23:52:08.843174 6752 mini_ranger.cc:68] Stopped Ranger
2026-09-28 23:52:08.843 UTC [18585] LOG: received smart shutdown request
2026-09-28 23:52:08.847 UTC [18585] LOG: background worker "logical replication launcher" (PID 18594) exited with exit code 1
2026-09-28 23:52:08.847 UTC [18589] LOG: shutting down
2026-09-28 23:52:09.029 UTC [18585] LOG: database system is shut down
2026-09-28T23:52:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47396 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:52:09 dist-test-slave-p5bw krb5kdc[20095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:52:09 dist-test-slave-p5bw krb5kdc[20095](info): set up 2 sockets
Sep 28 23:52:09 dist-test-slave-p5bw krb5kdc[20095](info): commencing operation
krb5kdc: starting...
W20260928 23:52:10.205593 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:52:10 dist-test-slave-p5bw krb5kdc[20095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:52:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:52:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:52:10.333278 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:36255
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:38883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:36255,127.6.152.61:35729,127.6.152.60:45563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:52:10.439656 20108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:10.439908 20108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:10.439949 20108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:10.443794 20108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:52:10.443843 20108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:10.443862 20108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:10.443878 20108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:52:10.443893 20108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:52:10.448865 20108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:38883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:36255,127.6.152.61:35729,127.6.152.60:45563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:36255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:10.449779 20108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:10.450690 20108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:10.458877 20114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:10.458851 20113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:10.458858 20116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:10.458861 20108 server_base.cc:1034] running on GCE node
I20260928 23:52:10.459643 20108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:10.460594 20108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:10.461755 20108 hybrid_clock.cc:648] HybridClock initialized: now 1790639530461751 us; error 57 us; skew 500 ppm
Sep 28 23:52:10 dist-test-slave-p5bw krb5kdc[20095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639530, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:10.464470 20108 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:52:10.465431 20108 webserver.cc:460] Webserver started at http://127.6.152.62:43859/ using document root <none> and password file <none>
I20260928 23:52:10.465976 20108 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:10.466037 20108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:10.466298 20108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:52:10.468238 20108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f"
format_stamp: "Formatted at 2026-09-28 23:52:10 on dist-test-slave-p5bw"
server_key: "c5a538b4a60ff3a532cd472e48c60c44"
server_key_iv: "cd9b7c003117e19da6ea1757cfb26ecb"
server_key_version: "encryptionkey@0"
I20260928 23:52:10.468801 20108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f"
format_stamp: "Formatted at 2026-09-28 23:52:10 on dist-test-slave-p5bw"
server_key: "c5a538b4a60ff3a532cd472e48c60c44"
server_key_iv: "cd9b7c003117e19da6ea1757cfb26ecb"
server_key_version: "encryptionkey@0"
I20260928 23:52:10.472394 20108 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:52:10.474690 20123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:10.475883 20108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:52:10.476017 20108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f"
format_stamp: "Formatted at 2026-09-28 23:52:10 on dist-test-slave-p5bw"
server_key: "c5a538b4a60ff3a532cd472e48c60c44"
server_key_iv: "cd9b7c003117e19da6ea1757cfb26ecb"
server_key_version: "encryptionkey@0"
I20260928 23:52:10.476123 20108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:10.498451 20108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:10.501255 20108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:10.501400 20108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:10.508427 20108 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:36255
I20260928 23:52:10.508451 20175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:36255 every 8 connection(s)
I20260928 23:52:10.509461 20108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:52:10.512018 20176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:52:10.515619 20176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36255 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:52:10.519192 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 20108
I20260928 23:52:10.519337 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:52:10.519610 6752 external_mini_cluster.cc:1428] Setting key ef8f129e8c25d98f18e76d0462ec266e
Sep 28 23:52:10 dist-test-slave-p5bw krb5kdc[20095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639530, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.152.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.61@KRBTEST.COM" created.
I20260928 23:52:10.531509 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.517712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60910 (local address 127.6.152.62:36255)
0928 23:52:10.518139 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:10.518157 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:10.518318 (+ 161us) server_negotiation.cc:407] Connection header received
0928 23:52:10.519446 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:10.519451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:10.519644 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:10.519836 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:10.520988 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.522374 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.523392 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.524028 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.524202 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:10.526725 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:10.526734 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:10.526746 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:10.526776 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:10.528613 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:10.529091 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:10.529095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:10.529096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:10.529237 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:10.529545 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:10.529547 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:10.529548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:10.529902 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:10.530000 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:10.530403 (+ 403us) server_negotiation.cc:299] Negotiation successful
0928 23:52:10.530583 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":312,"spinlock_wait_cycles":36224,"thread_start_us":162,"threads_started":1}
I20260928 23:52:10.531517 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.517712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36255 (local address 127.0.0.1:60910)
0928 23:52:10.518176 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.518212 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:10.519091 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:10.519875 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:10.519883 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:10.520377 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:10.520795 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:10.520818 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.522507 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.522510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:10.523267 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:10.523277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.524186 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.524190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:10.524312 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:10.524936 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:10.524951 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:10.526564 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:10.528710 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:10.528717 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:10.528725 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:10.528997 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:10.529341 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:10.529345 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:10.529347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:10.529458 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:10.529998 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:10.530003 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:10.530118 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 23:52:10.530304 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 23:52:10.530727 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":148,"thread_start_us":113,"threads_started":1}
I20260928 23:52:10.534214 20176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:52:10.535034 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.534680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:35729 (local address 127.0.0.1:32786)
0928 23:52:10.534842 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.534934 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.61:35729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 23:52:10.535269 20124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.152.61:35729: connect: Connection refused (error 111)
I20260928 23:52:10.536211 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.535870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:35729 (local address 127.0.0.1:32788)
0928 23:52:10.536016 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.536127 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.61:35729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 23:52:10.536403 20176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.61:35729: Network error: Client connection negotiation failed: client connection to 127.6.152.61:35729: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.61.keytab.
Entry for principal kudu/127.6.152.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.61.keytab.
I20260928 23:52:10.549067 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.61:35729
--webserver_interface=127.6.152.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.61.keytab
--principal=kudu/127.6.152.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:38883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:36255,127.6.152.61:35729,127.6.152.60:45563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:52:10.585687 20176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 } attempt: 1
I20260928 23:52:10.586620 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.586251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:35729 (local address 127.0.0.1:32792)
0928 23:52:10.586434 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.586523 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.61:35729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 23:52:10.587423 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.587168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:35729 (local address 127.0.0.1:32808)
0928 23:52:10.587287 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.587350 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.61:35729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 23:52:10.587584 20176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.61:35729: Network error: Client connection negotiation failed: client connection to 127.6.152.61:35729: connect: Connection refused (error 111)
I20260928 23:52:10.655759 20176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 } attempt: 2
I20260928 23:52:10.656603 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.656272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:35729 (local address 127.0.0.1:32810)
0928 23:52:10.656444 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.656512 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.61:35729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 23:52:10.657384 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.657148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:35729 (local address 127.0.0.1:32818)
0928 23:52:10.657263 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.657311 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.61:35729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 23:52:10.657536 20176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.61:35729: Network error: Client connection negotiation failed: client connection to 127.6.152.61:35729: connect: Connection refused (error 111)
W20260928 23:52:10.661365 20183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:10.661594 20183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:10.661634 20183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:10.665675 20183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:52:10.665720 20183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:10.665740 20183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:10.665757 20183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:52:10.665808 20183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:52:10.671113 20183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:38883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:36255,127.6.152.61:35729,127.6.152.60:45563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.61.keytab
--principal=kudu/127.6.152.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.61:35729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.152.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:10.672119 20183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:10.673184 20183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:10.681077 20191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:10.681097 20188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:10.681085 20189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:10.681496 20183 server_base.cc:1034] running on GCE node
I20260928 23:52:10.682013 20183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:10.683053 20183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:10.684260 20183 hybrid_clock.cc:648] HybridClock initialized: now 1790639530684246 us; error 29 us; skew 500 ppm
Sep 28 23:52:10 dist-test-slave-p5bw krb5kdc[20095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639530, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:10.686925 20183 init.cc:377] Logged in from keytab as kudu/127.6.152.61@KRBTEST.COM (short username kudu)
I20260928 23:52:10.687831 20183 webserver.cc:460] Webserver started at http://127.6.152.61:37655/ using document root <none> and password file <none>
I20260928 23:52:10.688393 20183 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:10.688490 20183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:10.688737 20183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:52:10.690840 20183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-1/data/instance:
uuid: "89b8166fa7d544478f0612b533fd22d4"
format_stamp: "Formatted at 2026-09-28 23:52:10 on dist-test-slave-p5bw"
server_key: "9da4547b8baccd53f59d02ed7a099d5b"
server_key_iv: "7502d33cc15917b1685ff92b7dad2792"
server_key_version: "encryptionkey@0"
I20260928 23:52:10.691401 20183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-1/wal/instance:
uuid: "89b8166fa7d544478f0612b533fd22d4"
format_stamp: "Formatted at 2026-09-28 23:52:10 on dist-test-slave-p5bw"
server_key: "9da4547b8baccd53f59d02ed7a099d5b"
server_key_iv: "7502d33cc15917b1685ff92b7dad2792"
server_key_version: "encryptionkey@0"
I20260928 23:52:10.695212 20183 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:52:10.697662 20198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:10.698990 20183 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:52:10.699187 20183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-1/wal
uuid: "89b8166fa7d544478f0612b533fd22d4"
format_stamp: "Formatted at 2026-09-28 23:52:10 on dist-test-slave-p5bw"
server_key: "9da4547b8baccd53f59d02ed7a099d5b"
server_key_iv: "7502d33cc15917b1685ff92b7dad2792"
server_key_version: "encryptionkey@0"
I20260928 23:52:10.699314 20183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:10.717600 20183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:10.720373 20183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:10.720585 20183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:10.727728 20183 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.61:35729
I20260928 23:52:10.727746 20250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.61:35729 every 8 connection(s)
I20260928 23:52:10.728696 20183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-1/data/info.pb
I20260928 23:52:10.731449 20251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:52:10.735102 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 20183
I20260928 23:52:10.735261 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-1/wal/instance
I20260928 23:52:10.735555 6752 external_mini_cluster.cc:1428] Setting key b78e7e51a186e779dfb728c75023b771
I20260928 23:52:10.735199 20251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36255 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.152.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.60@KRBTEST.COM" created.
Sep 28 23:52:10 dist-test-slave-p5bw krb5kdc[20095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639530, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.61@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:52:10.748690 20176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 } attempt: 3
I20260928 23:52:10.753167 20254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.737617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36255 (local address 127.0.0.1:60912)
0928 23:52:10.738085 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.738137 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:10.738989 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:10.739556 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:10.739562 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:10.739932 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:10.740647 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:10.740681 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.741541 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.741578 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:10.743033 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:10.743046 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.743601 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.743608 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:10.743805 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:10.744411 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:10.744427 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:10.747886 (+ 3459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:10.750003 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:10.750009 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:10.750024 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:10.750408 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:10.750723 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:10.750726 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:10.750728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:10.750896 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:10.751386 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:10.751392 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:10.751681 (+ 289us) client_negotiation.cc:769] Sending connection context
0928 23:52:10.751880 (+ 199us) client_negotiation.cc:240] Negotiation successful
0928 23:52:10.752162 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":146,"threads_started":1}
I20260928 23:52:10.754251 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.737264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60912 (local address 127.6.152.62:36255)
0928 23:52:10.737437 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:10.737443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:10.738262 (+ 819us) server_negotiation.cc:407] Connection header received
0928 23:52:10.739201 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:10.739208 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:10.739276 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:10.739389 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:10.740870 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.741408 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.743208 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.743436 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.743520 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:10.748079 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:10.748082 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:10.748084 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:10.748110 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:10.749877 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:10.750521 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:10.750528 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:10.750532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:10.750629 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:10.751012 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:10.751016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:10.751018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:10.751236 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:10.751308 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:10.753132 (+ 1824us) server_negotiation.cc:299] Negotiation successful
0928 23:52:10.753267 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 23:52:10 dist-test-slave-p5bw krb5kdc[20095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639530, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for kudu/127.6.152.61@KRBTEST.COM
I20260928 23:52:10.755681 20251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:52:10.760336 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.749171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:35729 (local address 127.0.0.1:32820)
0928 23:52:10.749384 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.749399 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:10.749485 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:10.750214 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:10.750217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:10.750384 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:10.750526 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:10.750536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.751911 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.751917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:10.752921 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:10.752933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.753844 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.753848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:10.753967 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:10.754463 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:10.754477 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:10.756247 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:10.758569 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:10.758572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:10.758573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:10.758778 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:10.759128 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:10.759132 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:10.759134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:10.759228 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:10.759833 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:10.759837 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:10.759955 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 23:52:10.760092 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 23:52:10.760209 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260928 23:52:10.760951 20256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.749303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32820 (local address 127.6.152.61:35729)
0928 23:52:10.749758 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:10.749772 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:10.749799 (+ 27us) server_negotiation.cc:407] Connection header received
0928 23:52:10.749845 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:10.749854 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:10.750065 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:10.750208 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:10.750652 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.751779 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.753048 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.753714 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.753842 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:10.756403 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:10.756412 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:10.756425 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:10.756444 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:10.758465 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:10.758888 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:10.758894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:10.758896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:10.758982 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:10.759328 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:10.759333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:10.759335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:10.759642 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:10.759952 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:10.760182 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 23:52:10.760356 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":143,"threads_started":1}
Sep 28 23:52:10 dist-test-slave-p5bw krb5kdc[20095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639530, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.61@KRBTEST.COM for kudu/127.6.152.61@KRBTEST.COM
I20260928 23:52:10.763182 20176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.60.keytab.
Entry for principal kudu/127.6.152.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.60.keytab.
I20260928 23:52:10.764002 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.763718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:45563 (local address 127.0.0.1:32838)
0928 23:52:10.763840 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.763910 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:45563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 23:52:10.764863 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.764526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:45563 (local address 127.0.0.1:32850)
0928 23:52:10.764714 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.764768 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:45563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260928 23:52:10.764902 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.60:45563
--webserver_interface=127.6.152.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.60.keytab
--principal=kudu/127.6.152.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:38883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:36255,127.6.152.61:35729,127.6.152.60:45563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:52:10.765034 20176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.60:45563: Network error: Client connection negotiation failed: client connection to 127.6.152.60:45563: connect: Connection refused (error 111)
I20260928 23:52:10.765576 20254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.756184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:35729 (local address 127.0.0.1:32828)
0928 23:52:10.756356 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.756371 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:10.756434 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:10.756864 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:10.756868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:10.757039 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:10.757156 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:10.757162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.758254 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.758257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:10.758960 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:10.758969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.759481 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.759485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:10.759581 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:10.760060 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:10.760079 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:10.761638 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:10.763715 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:10.763718 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:10.763720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:10.763970 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:10.764446 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:10.764453 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:10.764458 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:10.764557 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:10.765056 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:10.765060 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:10.765150 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 23:52:10.765265 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 23:52:10.765404 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 23:52:10.765621 20257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.756297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32828 (local address 127.6.152.61:35729)
0928 23:52:10.756636 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:10.756646 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:10.756668 (+ 22us) server_negotiation.cc:407] Connection header received
0928 23:52:10.756694 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:10.756696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:10.756727 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:10.756795 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:10.757272 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.758130 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.759087 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.759372 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.759453 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:10.761823 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:10.761829 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:10.761832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:10.761860 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:10.763582 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:10.764081 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:10.764087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:10.764089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:10.764169 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:10.764665 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:10.764668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:10.764669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:10.764922 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:10.765021 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:10.765318 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 23:52:10.765494 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":245,"thread_start_us":131,"threads_started":1}
I20260928 23:52:10.766525 20251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:52:10.767222 20254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.766994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:45563 (local address 127.0.0.1:32852)
0928 23:52:10.767104 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.767170 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:45563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 23:52:10.767513 20200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.152.60:45563: connect: Connection refused (error 111)
I20260928 23:52:10.768347 20254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.768163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:45563 (local address 127.0.0.1:32864)
0928 23:52:10.768260 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.768301 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:45563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 23:52:10.768587 20251 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.60:45563: Network error: Client connection negotiation failed: client connection to 127.6.152.60:45563: connect: Connection refused (error 111)
I20260928 23:52:10.796204 20176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } attempt: 1
I20260928 23:52:10.797163 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.796866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:45563 (local address 127.0.0.1:32870)
0928 23:52:10.797021 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.797082 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:45563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 23:52:10.797936 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.797710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:45563 (local address 127.0.0.1:32880)
0928 23:52:10.797833 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.797884 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:45563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 23:52:10.798070 20176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.60:45563: Network error: Client connection negotiation failed: client connection to 127.6.152.60:45563: connect: Connection refused (error 111)
I20260928 23:52:10.817824 20251 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } attempt: 1
I20260928 23:52:10.818660 20254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.818380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:45563 (local address 127.0.0.1:32882)
0928 23:52:10.818526 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.818587 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:45563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 23:52:10.819375 20254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.819158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:45563 (local address 127.0.0.1:32886)
0928 23:52:10.819266 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.819323 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:45563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 23:52:10.819506 20251 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.60:45563: Network error: Client connection negotiation failed: client connection to 127.6.152.60:45563: connect: Connection refused (error 111)
I20260928 23:52:10.873229 20176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } attempt: 2
I20260928 23:52:10.874083 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.873776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:45563 (local address 127.0.0.1:32898)
0928 23:52:10.873946 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.874006 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:45563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 23:52:10.874771 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.874557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:45563 (local address 127.0.0.1:32910)
0928 23:52:10.874667 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.874717 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:45563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 23:52:10.874902 20176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.60:45563: Network error: Client connection negotiation failed: client connection to 127.6.152.60:45563: connect: Connection refused (error 111)
W20260928 23:52:10.875869 20258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:10.876144 20258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:10.876217 20258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:10.880295 20258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:52:10.880383 20258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:10.880424 20258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:10.880465 20258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:52:10.880504 20258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:52:10.885828 20258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:38883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:36255,127.6.152.61:35729,127.6.152.60:45563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.60.keytab
--principal=kudu/127.6.152.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.60:45563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.152.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:10.886893 20258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:10.887686 20251 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } attempt: 2
I20260928 23:52:10.888098 20258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:52:10.888558 20254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.888276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:45563 (local address 127.0.0.1:32922)
0928 23:52:10.888419 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.888486 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:45563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 23:52:10.889292 20254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.889076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:45563 (local address 127.0.0.1:32936)
0928 23:52:10.889186 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.889242 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:45563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 23:52:10.889422 20251 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.60:45563: Network error: Client connection negotiation failed: client connection to 127.6.152.60:45563: connect: Connection refused (error 111)
W20260928 23:52:10.896716 20264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:10.896740 20265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:10.896797 20258 server_base.cc:1034] running on GCE node
W20260928 23:52:10.896745 20267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:10.897535 20258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:10.898617 20258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:10.899789 20258 hybrid_clock.cc:648] HybridClock initialized: now 1790639530899773 us; error 27 us; skew 500 ppm
Sep 28 23:52:10 dist-test-slave-p5bw krb5kdc[20095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639530, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:10.902643 20258 init.cc:377] Logged in from keytab as kudu/127.6.152.60@KRBTEST.COM (short username kudu)
I20260928 23:52:10.903525 20258 webserver.cc:460] Webserver started at http://127.6.152.60:36781/ using document root <none> and password file <none>
I20260928 23:52:10.904088 20258 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:10.904146 20258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:10.904307 20258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:52:10.906267 20258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-2/data/instance:
uuid: "2ba5bec69b19423a89872a0debc0fd2b"
format_stamp: "Formatted at 2026-09-28 23:52:10 on dist-test-slave-p5bw"
server_key: "f993ffd86ed29ecbc389d550e9f5ac3e"
server_key_iv: "09b208710b4c8b071950e48c27449fdf"
server_key_version: "encryptionkey@0"
I20260928 23:52:10.906767 20258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-2/wal/instance:
uuid: "2ba5bec69b19423a89872a0debc0fd2b"
format_stamp: "Formatted at 2026-09-28 23:52:10 on dist-test-slave-p5bw"
server_key: "f993ffd86ed29ecbc389d550e9f5ac3e"
server_key_iv: "09b208710b4c8b071950e48c27449fdf"
server_key_version: "encryptionkey@0"
I20260928 23:52:10.910301 20258 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:52:10.912595 20274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:10.913863 20258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:52:10.914018 20258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-2/wal
uuid: "2ba5bec69b19423a89872a0debc0fd2b"
format_stamp: "Formatted at 2026-09-28 23:52:10 on dist-test-slave-p5bw"
server_key: "f993ffd86ed29ecbc389d550e9f5ac3e"
server_key_iv: "09b208710b4c8b071950e48c27449fdf"
server_key_version: "encryptionkey@0"
I20260928 23:52:10.914136 20258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:10.930589 20258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:10.933553 20258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:10.933769 20258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:10.940604 20258 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.60:45563
I20260928 23:52:10.940629 20326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.60:45563 every 8 connection(s)
I20260928 23:52:10.941589 20258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-2/data/info.pb
I20260928 23:52:10.944231 20327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:52:10.947846 20327 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36255 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:52:10.951745 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 20258
I20260928 23:52:10.951905 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639374546653-6752-0/minicluster-data/master-2/wal/instance
I20260928 23:52:10.952184 6752 external_mini_cluster.cc:1428] Setting key d3b9d5f244f8b4e1e9a3ff7ac3df8614
Sep 28 23:52:10 dist-test-slave-p5bw krb5kdc[20095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639530, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.60@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
Sep 28 23:52:10 dist-test-slave-p5bw krb5kdc[20095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.60@KRBTEST.COM
I20260928 23:52:10.961973 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.949714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60914 (local address 127.6.152.62:36255)
0928 23:52:10.949852 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:10.949856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:10.950606 (+ 750us) server_negotiation.cc:407] Connection header received
0928 23:52:10.951520 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:10.951524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:10.951580 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:10.951705 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:10.953065 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.953764 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.954882 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.955219 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.955343 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:10.958094 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:10.958097 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:10.958100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:10.958125 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:10.959695 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:10.960310 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:10.960314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:10.960316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:10.960443 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:10.960836 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:10.960839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:10.960841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:10.961039 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:10.961161 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:10.961672 (+ 511us) server_negotiation.cc:299] Negotiation successful
0928 23:52:10.961803 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:52:10.962741 20329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.950008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:36255 (local address 127.0.0.1:60914)
0928 23:52:10.950450 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.950492 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:10.951310 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:10.951892 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:10.951902 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:10.952362 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:10.952910 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:10.952938 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.953895 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.953899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:10.954750 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:10.954759 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.955376 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.955379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:10.955550 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:10.956126 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:10.956183 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:10.957957 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:10.959889 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:10.959895 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:10.959914 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:10.960164 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:10.960558 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:10.960561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:10.960564 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:10.960704 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:10.961157 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:10.961163 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:10.961410 (+ 247us) client_negotiation.cc:769] Sending connection context
0928 23:52:10.961620 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 23:52:10.961922 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":120,"threads_started":1}
I20260928 23:52:10.964164 20327 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:52:10.964654 20331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.952994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32940 (local address 127.6.152.60:45563)
0928 23:52:10.953420 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:10.953428 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:10.953450 (+ 22us) server_negotiation.cc:407] Connection header received
0928 23:52:10.953565 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:10.953569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:10.953728 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:10.953857 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:10.955769 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.956908 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.957942 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.958367 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.958449 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:10.960907 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:10.960915 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:10.960924 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:10.960944 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:10.962626 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:10.963141 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:10.963153 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:10.963154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:10.963270 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:10.963612 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:10.963619 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:10.963623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:10.963881 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:10.964020 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:10.964321 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 23:52:10.964466 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":156,"threads_started":1}
Sep 28 23:52:10 dist-test-slave-p5bw krb5kdc[20095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639530, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.60@KRBTEST.COM for kudu/127.6.152.61@KRBTEST.COM
Sep 28 23:52:10 dist-test-slave-p5bw krb5kdc[20095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:52:10.973366 20329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.964799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:35729 (local address 127.0.0.1:32840)
0928 23:52:10.965127 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.965149 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:10.965246 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:10.965518 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:10.965521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:10.965649 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:10.965749 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:10.965754 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.966711 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.966714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:10.967543 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:10.967555 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.968041 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.968044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:10.968116 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:10.968597 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:10.968610 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:10.970030 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:10.971862 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:10.971865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:10.971867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:10.972087 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:10.972407 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:10.972413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:10.972415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:10.972503 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:10.972989 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:10.972992 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:10.973064 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 23:52:10.973147 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 23:52:10.973263 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257}
I20260928 23:52:10.973658 20257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.964903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32840 (local address 127.6.152.61:35729)
0928 23:52:10.965045 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:10.965050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:10.965212 (+ 162us) server_negotiation.cc:407] Connection header received
0928 23:52:10.965325 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:10.965329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:10.965374 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:10.965476 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:10.965869 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.966569 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.967665 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.967933 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.968033 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:10.970142 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:10.970146 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:10.970150 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:10.970181 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:10.971746 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:10.972183 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:10.972190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:10.972194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:10.972280 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:10.972601 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:10.972605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:10.972607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:10.972887 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:10.972995 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:10.973229 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 23:52:10.973343 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:52:10.974035 20176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } attempt: 3
I20260928 23:52:10.974324 20327 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:52:10.975581 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.966854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60928 (local address 127.6.152.62:36255)
0928 23:52:10.966974 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:10.966977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:10.966989 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:52:10.967099 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:10.967100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:10.967131 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:10.967190 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:10.967892 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.968455 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.969375 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.969573 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.969650 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:10.971969 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:10.971972 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:10.971974 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:10.971993 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:10.973631 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:10.974121 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:10.974124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:10.974128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:10.974245 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:10.974555 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:10.974560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:10.974564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:10.974788 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:10.974987 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:10.975305 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 23:52:10.975416 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 23:52:10 dist-test-slave-p5bw krb5kdc[20095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639530, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for kudu/127.6.152.60@KRBTEST.COM
Sep 28 23:52:10 dist-test-slave-p5bw krb5kdc[20095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639530, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.60@KRBTEST.COM for kudu/127.6.152.60@KRBTEST.COM
I20260928 23:52:10.980551 20251 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } attempt: 3
Sep 28 23:52:10 dist-test-slave-p5bw krb5kdc[20095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.61@KRBTEST.COM
I20260928 23:52:10.983573 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.974428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:45563 (local address 127.0.0.1:32942)
0928 23:52:10.974562 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.974579 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:10.974658 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:10.975026 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:10.975031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:10.975196 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:10.975274 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:10.975279 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.976259 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.976262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:10.977169 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:10.977179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.977708 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.977711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:10.977792 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:10.978159 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:10.978169 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:10.979637 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:10.981768 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:10.981773 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:10.981776 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:10.982032 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:10.982415 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:10.982421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:10.982425 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:10.982518 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:10.983046 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:10.983052 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:10.983158 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 23:52:10.983274 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 23:52:10.983417 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 23:52:10.983604 20331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.974703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.6.152.60:45563)
0928 23:52:10.974798 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:10.974802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:10.974815 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:52:10.974852 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:10.974857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:10.974893 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:10.975002 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:10.975390 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.976133 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.977302 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.977593 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.977707 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:10.979813 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:10.979817 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:10.979820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:10.979844 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:10.981644 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:10.982202 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:10.982207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:10.982210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:10.982296 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:10.982642 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:10.982646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:10.982649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:10.982939 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:10.983036 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:10.983338 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 23:52:10.983438 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":38}
I20260928 23:52:10.985064 20329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.974741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:45563 (local address 127.0.0.1:32948)
0928 23:52:10.974832 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.974844 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:10.974909 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:10.975736 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:10.975739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:10.975888 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:10.975958 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:10.975963 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.977414 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.977421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:10.978213 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:10.978224 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.978868 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.978872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:10.978951 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:10.979389 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:10.979402 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:10.981078 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:10.983688 (+ 2610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:10.983694 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:10.983699 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:10.983927 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:10.984184 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:10.984187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:10.984189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:10.984271 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:10.984598 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:10.984601 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:10.984680 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 23:52:10.984763 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 23:52:10.984910 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 23:52:10.985590 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.974911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32948 (local address 127.6.152.60:45563)
0928 23:52:10.975542 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:10.975550 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:10.975562 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:52:10.975590 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:10.975593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:10.975628 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:10.976042 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:10.976098 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.977302 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.978334 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.978750 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.978872 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:10.981235 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:10.981238 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:10.981240 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:10.981259 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:10.983586 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:10.984027 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:10.984030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:10.984032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:10.984102 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:10.984355 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:10.984358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:10.984359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:10.984512 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:10.985215 (+ 703us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:10.985302 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 23:52:10.985416 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":572,"thread_start_us":128,"threads_started":1}
Sep 28 23:52:10 dist-test-slave-p5bw krb5kdc[20095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639530, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.61@KRBTEST.COM for kudu/127.6.152.60@KRBTEST.COM
I20260928 23:52:10.987447 20176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f: Bootstrap starting.
I20260928 23:52:10.988323 20257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.976649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32850 (local address 127.6.152.61:35729)
0928 23:52:10.976745 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:10.976749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:10.976831 (+ 82us) server_negotiation.cc:407] Connection header received
0928 23:52:10.976974 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:10.976977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:10.977013 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:10.977106 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:10.977797 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.978613 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.979949 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.980312 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.980455 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:10.984467 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:10.984471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:10.984473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:10.984499 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:10.986165 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:10.986869 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:10.986872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:10.986874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:10.986953 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:10.987247 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:10.987251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:10.987252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:10.987462 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:10.987637 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:10.987968 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 23:52:10.988069 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 23:52:10.988595 20327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba5bec69b19423a89872a0debc0fd2b: Bootstrap starting.
I20260928 23:52:10.990288 20176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f: Neither blocks nor log segments found. Creating new log.
I20260928 23:52:10.990938 20327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ba5bec69b19423a89872a0debc0fd2b: Neither blocks nor log segments found. Creating new log.
I20260928 23:52:10.991182 20176 log.cc:824] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:10.991642 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:52:10.991746 20327 log.cc:824] T 00000000000000000000000000000000 P 2ba5bec69b19423a89872a0debc0fd2b: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:10.993364 20176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f: No bootstrap required, opened a new log
I20260928 23:52:10.993656 20327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba5bec69b19423a89872a0debc0fd2b: No bootstrap required, opened a new log
I20260928 23:52:10.997624 20176 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36255 } } peers { permanent_uuid: "89b8166fa7d544478f0612b533fd22d4" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 } } peers { permanent_uuid: "2ba5bec69b19423a89872a0debc0fd2b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } }
I20260928 23:52:10.997845 20176 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:52:10.997713 20327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ba5bec69b19423a89872a0debc0fd2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36255 } } peers { permanent_uuid: "89b8166fa7d544478f0612b533fd22d4" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 } } peers { permanent_uuid: "2ba5bec69b19423a89872a0debc0fd2b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } }
I20260928 23:52:10.997908 20176 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2728bcc9c7a4a09a85c73f13c5ba30f, State: Initialized, Role: FOLLOWER
I20260928 23:52:10.997923 20327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ba5bec69b19423a89872a0debc0fd2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:52:10.997987 20327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ba5bec69b19423a89872a0debc0fd2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba5bec69b19423a89872a0debc0fd2b, State: Initialized, Role: FOLLOWER
I20260928 23:52:10.998404 20176 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36255 } } peers { permanent_uuid: "89b8166fa7d544478f0612b533fd22d4" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 } } peers { permanent_uuid: "2ba5bec69b19423a89872a0debc0fd2b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } }
I20260928 23:52:10.998637 20327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ba5bec69b19423a89872a0debc0fd2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36255 } } peers { permanent_uuid: "89b8166fa7d544478f0612b533fd22d4" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 } } peers { permanent_uuid: "2ba5bec69b19423a89872a0debc0fd2b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } }
I20260928 23:52:10.999770 20176 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:52:10.999974 20327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ba5bec69b19423a89872a0debc0fd2b [sys.catalog]: configured and running, proceeding with master startup.
W20260928 23:52:11.002774 20129 tablet.cc:2369] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:52:11.003036 20354 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36255 } } peers { permanent_uuid: "89b8166fa7d544478f0612b533fd22d4" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 } } peers { permanent_uuid: "2ba5bec69b19423a89872a0debc0fd2b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } } }
I20260928 23:52:11.003249 20354 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:52:11.003351 20354 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:52:11.003456 20354 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36255 } } peers { permanent_uuid: "89b8166fa7d544478f0612b533fd22d4" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 } } peers { permanent_uuid: "2ba5bec69b19423a89872a0debc0fd2b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } }
I20260928 23:52:11.005018 20301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ba5bec69b19423a89872a0debc0fd2b" is_pre_election: true
I20260928 23:52:11.005292 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.993780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60936 (local address 127.6.152.62:36255)
0928 23:52:10.993892 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:10.993897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:10.994592 (+ 695us) server_negotiation.cc:407] Connection header received
0928 23:52:10.994685 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:10.994688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:10.994730 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:10.994807 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:10.995706 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.996444 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.997638 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.997941 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.998046 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:11.000289 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:11.000296 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:11.000299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:11.000328 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:11.002553 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:11.003118 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:11.003122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:11.003124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:11.003206 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:11.003819 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:11.003822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:11.003824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:11.004096 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:11.004211 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:11.004987 (+ 776us) server_negotiation.cc:299] Negotiation successful
0928 23:52:11.005100 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:52:11.005307 20301 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2ba5bec69b19423a89872a0debc0fd2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2728bcc9c7a4a09a85c73f13c5ba30f in term 0.
I20260928 23:52:11.005946 20125 leader_election.cc:302] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ba5bec69b19423a89872a0debc0fd2b, b2728bcc9c7a4a09a85c73f13c5ba30f; no voters:
I20260928 23:52:11.006192 20354 leader_election.cc:288] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89b8166fa7d544478f0612b533fd22d4 (127.6.152.61:35729), 2ba5bec69b19423a89872a0debc0fd2b (127.6.152.60:45563)
I20260928 23:52:11.006292 20354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:52:11.007062 20343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ba5bec69b19423a89872a0debc0fd2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36255 } } peers { permanent_uuid: "89b8166fa7d544478f0612b533fd22d4" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 } } peers { permanent_uuid: "2ba5bec69b19423a89872a0debc0fd2b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } } }
I20260928 23:52:11.007190 20343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ba5bec69b19423a89872a0debc0fd2b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:52:11.007768 20254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.980954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:45563 (local address 127.0.0.1:32958)
0928 23:52:10.981124 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:10.981138 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:10.981227 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:10.982323 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:10.982327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:10.982505 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:10.982596 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:10.982604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.983788 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.983792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:10.984597 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:10.984607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.985990 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.985996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:10.986080 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:10.986485 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:10.986499 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:10.987809 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:11.005415 (+ 17606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:11.005421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:11.005423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:11.005659 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:11.006755 (+ 1096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:11.006761 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:11.006763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:11.006853 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:11.007275 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:11.007278 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:11.007375 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 23:52:11.007455 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 23:52:11.007597 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 23:52:11.008356 20333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.981353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32958 (local address 127.6.152.60:45563)
0928 23:52:10.981849 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:10.981859 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:10.981874 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:52:10.981906 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:10.981911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:10.981954 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:10.982712 (+ 758us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:10.982863 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.983658 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.984992 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.985861 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.992167 (+ 6306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:10.992280 (+ 113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:10.992282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:10.992285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:10.992302 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:11.005285 (+ 12983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:11.006539 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:11.006543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:11.006544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:11.006634 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:11.006964 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:11.006966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:11.006968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:11.007171 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:11.008005 (+ 834us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:11.008077 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 23:52:11.008188 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":419,"thread_start_us":95,"threads_started":1}
W20260928 23:52:11.010742 20366 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ba5bec69b19423a89872a0debc0fd2b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 23:52:11.007675 20258 master_runner.cc:186] Error getting master registration for 127.6.152.61:35729: OK, instance_id { permanent_uuid: "89b8166fa7d544478f0612b533fd22d4" instance_seqno: 1790639530725701 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260928 23:52:11.010951 20366 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:52:11.011031 20258 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:52:11.011183 20251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89b8166fa7d544478f0612b533fd22d4: Bootstrap starting.
I20260928 23:52:11.006351 20108 master_runner.cc:186] Error getting master registration for 127.6.152.61:35729: OK, instance_id { permanent_uuid: "89b8166fa7d544478f0612b533fd22d4" instance_seqno: 1790639530725701 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:52:11.011684 20108 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:52:11.006358 20354 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:52:11.012796 20354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:52:11.012893 20257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:10.994168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32858 (local address 127.6.152.61:35729)
0928 23:52:10.994329 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:10.994333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:10.994849 (+ 516us) server_negotiation.cc:407] Connection header received
0928 23:52:10.994999 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:10.995002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:10.995039 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:10.995085 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:10.995991 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.996715 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.998091 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:10.998388 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:10.998463 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:11.000520 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:11.000524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:11.000527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:11.000554 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:11.002244 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:11.009784 (+ 7540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:11.009787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:11.009789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:11.009887 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:11.011999 (+ 2112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:11.012002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:11.012004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:11.012239 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:11.012373 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:11.012622 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 23:52:11.012733 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 23:52:11.013674 20251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89b8166fa7d544478f0612b533fd22d4: Neither blocks nor log segments found. Creating new log.
I20260928 23:52:11.014122 20354 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36255 } } peers { permanent_uuid: "89b8166fa7d544478f0612b533fd22d4" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 } } peers { permanent_uuid: "2ba5bec69b19423a89872a0debc0fd2b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } }
I20260928 23:52:11.014377 20251 log.cc:824] T 00000000000000000000000000000000 P 89b8166fa7d544478f0612b533fd22d4: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:11.015301 20354 leader_election.cc:288] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [CANDIDATE]: Term 1 election: Requested vote from peers 89b8166fa7d544478f0612b533fd22d4 (127.6.152.61:35729), 2ba5bec69b19423a89872a0debc0fd2b (127.6.152.60:45563)
I20260928 23:52:11.015350 20301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ba5bec69b19423a89872a0debc0fd2b"
I20260928 23:52:11.015465 20301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ba5bec69b19423a89872a0debc0fd2b [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:52:11.016640 20301 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2ba5bec69b19423a89872a0debc0fd2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2728bcc9c7a4a09a85c73f13c5ba30f in term 1.
I20260928 23:52:11.016984 20125 leader_election.cc:302] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ba5bec69b19423a89872a0debc0fd2b, b2728bcc9c7a4a09a85c73f13c5ba30f; no voters:
I20260928 23:52:11.017186 20354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:52:11.017525 20354 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [term 1 LEADER]: Becoming Leader. State: Replica: b2728bcc9c7a4a09a85c73f13c5ba30f, State: Running, Role: LEADER
W20260928 23:52:11.017896 20365 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:52:11.017961 20365 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:52:11.017990 20251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89b8166fa7d544478f0612b533fd22d4: No bootstrap required, opened a new log
I20260928 23:52:11.018127 20354 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36255 } } peers { permanent_uuid: "89b8166fa7d544478f0612b533fd22d4" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 } } peers { permanent_uuid: "2ba5bec69b19423a89872a0debc0fd2b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } }
I20260928 23:52:11.020036 20342 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2728bcc9c7a4a09a85c73f13c5ba30f. Latest consensus state: current_term: 1 leader_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36255 } } peers { permanent_uuid: "89b8166fa7d544478f0612b533fd22d4" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 } } peers { permanent_uuid: "2ba5bec69b19423a89872a0debc0fd2b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } } }
I20260928 23:52:11.020182 20342 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [sys.catalog]: This master's current role is: LEADER
I20260928 23:52:11.022622 20251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89b8166fa7d544478f0612b533fd22d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36255 } } peers { permanent_uuid: "89b8166fa7d544478f0612b533fd22d4" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 } } peers { permanent_uuid: "2ba5bec69b19423a89872a0debc0fd2b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } }
I20260928 23:52:11.022856 20251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89b8166fa7d544478f0612b533fd22d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:52:11.022918 20251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89b8166fa7d544478f0612b533fd22d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89b8166fa7d544478f0612b533fd22d4, State: Initialized, Role: FOLLOWER
I20260928 23:52:11.023424 20251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89b8166fa7d544478f0612b533fd22d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36255 } } peers { permanent_uuid: "89b8166fa7d544478f0612b533fd22d4" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 } } peers { permanent_uuid: "2ba5bec69b19423a89872a0debc0fd2b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } }
I20260928 23:52:11.023983 20373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89b8166fa7d544478f0612b533fd22d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36255 } } peers { permanent_uuid: "89b8166fa7d544478f0612b533fd22d4" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 } } peers { permanent_uuid: "2ba5bec69b19423a89872a0debc0fd2b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } } }
I20260928 23:52:11.024202 20373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89b8166fa7d544478f0612b533fd22d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:52:11.024842 20251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89b8166fa7d544478f0612b533fd22d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:52:11.024885 20225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89b8166fa7d544478f0612b533fd22d4" is_pre_election: true
I20260928 23:52:11.025095 20224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89b8166fa7d544478f0612b533fd22d4"
I20260928 23:52:11.025144 20225 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 89b8166fa7d544478f0612b533fd22d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2728bcc9c7a4a09a85c73f13c5ba30f in term 0.
I20260928 23:52:11.025777 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:11.002736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32960 (local address 127.6.152.60:45563)
0928 23:52:11.012045 (+ 9309us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:11.012050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:11.012063 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:52:11.012097 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:11.012100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:11.012144 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:11.012213 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:11.013649 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:11.014975 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:11.016231 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:11.016572 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:11.016649 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:11.019339 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:11.019369 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:11.019373 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:11.019401 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:11.021909 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:11.022641 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:11.022645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:11.022647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:11.022784 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:11.023538 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:11.023541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:11.023543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:11.023974 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:11.024250 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:11.025503 (+ 1253us) server_negotiation.cc:299] Negotiation successful
0928 23:52:11.025630 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9231}
I20260928 23:52:11.033526 20372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:52:11.034474 20372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260928 23:52:11.037911 20384 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 89b8166fa7d544478f0612b533fd22d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:52:11.038051 20384 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:52:11.049450 20301 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2ba5bec69b19423a89872a0debc0fd2b [term 1 FOLLOWER]: Refusing update from remote peer b2728bcc9c7a4a09a85c73f13c5ba30f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:52:11.050061 20354 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ba5bec69b19423a89872a0debc0fd2b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:52:11.055862 20343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ba5bec69b19423a89872a0debc0fd2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2728bcc9c7a4a09a85c73f13c5ba30f. Latest consensus state: current_term: 1 leader_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36255 } } peers { permanent_uuid: "89b8166fa7d544478f0612b533fd22d4" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 } } peers { permanent_uuid: "2ba5bec69b19423a89872a0debc0fd2b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } } }
I20260928 23:52:11.055986 20343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ba5bec69b19423a89872a0debc0fd2b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:52:11.056779 20224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89b8166fa7d544478f0612b533fd22d4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:52:11.058249 20224 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 89b8166fa7d544478f0612b533fd22d4 [term 1 FOLLOWER]: Refusing update from remote peer b2728bcc9c7a4a09a85c73f13c5ba30f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:52:11.058789 20354 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89b8166fa7d544478f0612b533fd22d4" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:52:11.062203 20373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89b8166fa7d544478f0612b533fd22d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2728bcc9c7a4a09a85c73f13c5ba30f. Latest consensus state: current_term: 1 leader_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36255 } } peers { permanent_uuid: "89b8166fa7d544478f0612b533fd22d4" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 } } peers { permanent_uuid: "2ba5bec69b19423a89872a0debc0fd2b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } } }
I20260928 23:52:11.062331 20373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89b8166fa7d544478f0612b533fd22d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:52:11.065863 20354 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36255 } } peers { permanent_uuid: "89b8166fa7d544478f0612b533fd22d4" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 } } peers { permanent_uuid: "2ba5bec69b19423a89872a0debc0fd2b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } } }
I20260928 23:52:11.065989 20354 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [sys.catalog]: This master's current role is: LEADER
I20260928 23:52:11.066625 20386 mvcc.cc:205] Tried to move back new op lower bound from 7334459519165960192 to 7334459519053115392. Current Snapshot: MvccSnapshot[applied={T|T < 7334459519165960192}]
I20260928 23:52:11.067220 20343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ba5bec69b19423a89872a0debc0fd2b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36255 } } peers { permanent_uuid: "89b8166fa7d544478f0612b533fd22d4" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 } } peers { permanent_uuid: "2ba5bec69b19423a89872a0debc0fd2b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } } }
I20260928 23:52:11.067341 20343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ba5bec69b19423a89872a0debc0fd2b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:52:11.067703 20387 mvcc.cc:205] Tried to move back new op lower bound from 7334459519165960192 to 7334459519053115392. Current Snapshot: MvccSnapshot[applied={T|T < 7334459519165960192}]
I20260928 23:52:11.070281 20342 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36255 } } peers { permanent_uuid: "89b8166fa7d544478f0612b533fd22d4" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 } } peers { permanent_uuid: "2ba5bec69b19423a89872a0debc0fd2b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } } }
I20260928 23:52:11.070399 20342 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f [sys.catalog]: This master's current role is: LEADER
I20260928 23:52:11.071425 20373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89b8166fa7d544478f0612b533fd22d4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2728bcc9c7a4a09a85c73f13c5ba30f" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36255 } } peers { permanent_uuid: "89b8166fa7d544478f0612b533fd22d4" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 35729 } } peers { permanent_uuid: "2ba5bec69b19423a89872a0debc0fd2b" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 45563 } } }
I20260928 23:52:11.071542 20373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89b8166fa7d544478f0612b533fd22d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:52:11.071637 20372 catalog_manager.cc:1347] Generated new cluster ID: 493fe0ceccdb46c8ab1a491a64ee048d
I20260928 23:52:11.071702 20372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:52:11.072171 20389 mvcc.cc:205] Tried to move back new op lower bound from 7334459519165960192 to 7334459519053115392. Current Snapshot: MvccSnapshot[applied={T|T < 7334459519165960192}]
I20260928 23:52:11.092273 20372 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:52:11.093609 20372 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:52:11.116685 20372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2728bcc9c7a4a09a85c73f13c5ba30f: Generated new TSK 0
I20260928 23:52:11.117763 20372 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:52:12.012328 20366 catalog_manager.cc:1259] Loaded cluster ID: 493fe0ceccdb46c8ab1a491a64ee048d
I20260928 23:52:12.012413 20366 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2ba5bec69b19423a89872a0debc0fd2b: loading cluster ID for follower catalog manager: success
I20260928 23:52:12.014091 20366 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2ba5bec69b19423a89872a0debc0fd2b: acquiring CA information for follower catalog manager: success
I20260928 23:52:12.015244 20366 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2ba5bec69b19423a89872a0debc0fd2b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:52:12.039387 20384 catalog_manager.cc:1259] Loaded cluster ID: 493fe0ceccdb46c8ab1a491a64ee048d
I20260928 23:52:12.039467 20384 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 89b8166fa7d544478f0612b533fd22d4: loading cluster ID for follower catalog manager: success
I20260928 23:52:12.041177 20384 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 89b8166fa7d544478f0612b533fd22d4: acquiring CA information for follower catalog manager: success
I20260928 23:52:12.042348 20384 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 89b8166fa7d544478f0612b533fd22d4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:52:12.184979 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 20108
I20260928 23:52:12.191347 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 20183
I20260928 23:52:12.196758 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 20258
2026-09-28T23:52:12Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3047 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:52:12 dist-test-slave-p5bw krb5kdc[20401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:52:12 dist-test-slave-p5bw krb5kdc[20401](info): set up 2 sockets
Sep 28 23:52:12 dist-test-slave-p5bw krb5kdc[20401](info): commencing operation
krb5kdc: starting...
W20260928 23:52:13.225720 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:52:13 dist-test-slave-p5bw krb5kdc[20401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:52:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:52:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:52:13.352898 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:43531
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:43531
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:52:13.466759 20414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:13.467031 20414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:13.467079 20414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:13.471191 20414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:52:13.471249 20414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:13.471269 20414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:13.471287 20414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:52:13.471303 20414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:52:13.476890 20414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:43531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:43531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:13.477883 20414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:13.478819 20414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:13.487317 20420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:13.487335 20419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:13.487421 20414 server_base.cc:1034] running on GCE node
W20260928 23:52:13.487339 20422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:13.488180 20414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:13.489378 20414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:13.490553 20414 hybrid_clock.cc:648] HybridClock initialized: now 1790639533490538 us; error 28 us; skew 500 ppm
Sep 28 23:52:13 dist-test-slave-p5bw krb5kdc[20401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639533, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:13.493450 20414 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:52:13.494311 20414 webserver.cc:460] Webserver started at http://127.6.152.62:39245/ using document root <none> and password file <none>
I20260928 23:52:13.494832 20414 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:13.494890 20414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:13.495069 20414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:52:13.497120 20414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "cb27163e8ea34799bf8375ab824b04a2"
format_stamp: "Formatted at 2026-09-28 23:52:13 on dist-test-slave-p5bw"
server_key: "dd7f685ea49da06f28f13c3ed5c3a31a"
server_key_iv: "4693233b9d462505e29ddaf7e443d322"
server_key_version: "encryptionkey@0"
I20260928 23:52:13.497641 20414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "cb27163e8ea34799bf8375ab824b04a2"
format_stamp: "Formatted at 2026-09-28 23:52:13 on dist-test-slave-p5bw"
server_key: "dd7f685ea49da06f28f13c3ed5c3a31a"
server_key_iv: "4693233b9d462505e29ddaf7e443d322"
server_key_version: "encryptionkey@0"
I20260928 23:52:13.501274 20414 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:52:13.503603 20429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:13.504930 20414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:52:13.505025 20414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "cb27163e8ea34799bf8375ab824b04a2"
format_stamp: "Formatted at 2026-09-28 23:52:13 on dist-test-slave-p5bw"
server_key: "dd7f685ea49da06f28f13c3ed5c3a31a"
server_key_iv: "4693233b9d462505e29ddaf7e443d322"
server_key_version: "encryptionkey@0"
I20260928 23:52:13.505111 20414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:13.517062 20414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:13.519922 20414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:13.520128 20414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:13.527598 20414 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:43531
I20260928 23:52:13.527616 20481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:43531 every 8 connection(s)
I20260928 23:52:13.528541 20414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:52:13.528988 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 20414
I20260928 23:52:13.529109 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:52:13.529431 6752 external_mini_cluster.cc:1428] Setting key f75542748eb78a4502db1614ffe98930
I20260928 23:52:13.531685 20482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:52:13.536906 20482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb27163e8ea34799bf8375ab824b04a2: Bootstrap starting.
Sep 28 23:52:13 dist-test-slave-p5bw krb5kdc[20401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:52:13.539028 20482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb27163e8ea34799bf8375ab824b04a2: Neither blocks nor log segments found. Creating new log.
I20260928 23:52:13.539716 20482 log.cc:824] T 00000000000000000000000000000000 P cb27163e8ea34799bf8375ab824b04a2: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:13.541635 20482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb27163e8ea34799bf8375ab824b04a2: No bootstrap required, opened a new log
I20260928 23:52:13.543493 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:13.530846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33880 (local address 127.6.152.62:43531)
0928 23:52:13.531275 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:13.531290 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:13.531323 (+ 33us) server_negotiation.cc:407] Connection header received
0928 23:52:13.532032 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:13.532049 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:13.532366 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:13.532709 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:13.533540 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:13.534528 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:13.535415 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:13.535885 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:13.536068 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:13.538648 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:13.538653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:13.538666 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:13.538692 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:13.540649 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:13.541124 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:13.541130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:13.541147 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:13.541252 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:13.541556 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:13.541559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:13.541561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:13.542013 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:13.542213 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:13.542535 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 23:52:13.542814 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":131,"threads_started":1}
I20260928 23:52:13.545133 20482 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb27163e8ea34799bf8375ab824b04a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb27163e8ea34799bf8375ab824b04a2" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 43531 } }
I20260928 23:52:13.545416 20482 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb27163e8ea34799bf8375ab824b04a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:52:13.545501 20482 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb27163e8ea34799bf8375ab824b04a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb27163e8ea34799bf8375ab824b04a2, State: Initialized, Role: FOLLOWER
I20260928 23:52:13.545910 20482 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb27163e8ea34799bf8375ab824b04a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb27163e8ea34799bf8375ab824b04a2" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 43531 } }
I20260928 23:52:13.546027 20482 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb27163e8ea34799bf8375ab824b04a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:52:13.546106 20482 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb27163e8ea34799bf8375ab824b04a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:52:13.546211 20482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb27163e8ea34799bf8375ab824b04a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:52:13.546377 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:52:13.547106 20482 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb27163e8ea34799bf8375ab824b04a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb27163e8ea34799bf8375ab824b04a2" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 43531 } }
I20260928 23:52:13.547432 20482 leader_election.cc:302] T 00000000000000000000000000000000 P cb27163e8ea34799bf8375ab824b04a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb27163e8ea34799bf8375ab824b04a2; no voters:
I20260928 23:52:13.547755 20482 leader_election.cc:288] T 00000000000000000000000000000000 P cb27163e8ea34799bf8375ab824b04a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:52:13.547821 20487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb27163e8ea34799bf8375ab824b04a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:52:13.548032 20487 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb27163e8ea34799bf8375ab824b04a2 [term 1 LEADER]: Becoming Leader. State: Replica: cb27163e8ea34799bf8375ab824b04a2, State: Running, Role: LEADER
I20260928 23:52:13.548321 20487 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb27163e8ea34799bf8375ab824b04a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb27163e8ea34799bf8375ab824b04a2" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 43531 } }
I20260928 23:52:13.548986 20482 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb27163e8ea34799bf8375ab824b04a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:52:13.549978 20489 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb27163e8ea34799bf8375ab824b04a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb27163e8ea34799bf8375ab824b04a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb27163e8ea34799bf8375ab824b04a2" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 43531 } } }
I20260928 23:52:13.550032 20488 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb27163e8ea34799bf8375ab824b04a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb27163e8ea34799bf8375ab824b04a2. Latest consensus state: current_term: 1 leader_uuid: "cb27163e8ea34799bf8375ab824b04a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb27163e8ea34799bf8375ab824b04a2" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 43531 } } }
I20260928 23:52:13.550104 20489 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb27163e8ea34799bf8375ab824b04a2 [sys.catalog]: This master's current role is: LEADER
I20260928 23:52:13.550128 20488 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb27163e8ea34799bf8375ab824b04a2 [sys.catalog]: This master's current role is: LEADER
I20260928 23:52:13.550470 20496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:52:13.553028 20496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:52:13.557971 20496 catalog_manager.cc:1347] Generated new cluster ID: 57372c5188a441eba3228323e85342db
I20260928 23:52:13.558030 20496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:52:13.582885 20496 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:52:13.584252 20496 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:52:13.594736 20496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb27163e8ea34799bf8375ab824b04a2: Generated new TSK 0
I20260928 23:52:13.595558 20496 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:52:13.601980 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 20414
2026-09-28T23:52:13Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1403 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:52:13 dist-test-slave-p5bw krb5kdc[20509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:52:13 dist-test-slave-p5bw krb5kdc[20509](info): set up 2 sockets
Sep 28 23:52:13 dist-test-slave-p5bw krb5kdc[20509](info): commencing operation
krb5kdc: starting...
W20260928 23:52:14.637562 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:52:14 dist-test-slave-p5bw krb5kdc[20509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:52:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:52:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:52:14.760006 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:38395
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:38395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:52:14.868418 20522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:14.868731 20522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:14.868857 20522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:14.873057 20522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:52:14.873144 20522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:14.873181 20522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:14.873240 20522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:52:14.873277 20522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:52:14.878975 20522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:38395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:38395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:14.880093 20522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:14.881361 20522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:14.889835 20528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:14.889837 20527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:14.889837 20530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:14.889915 20522 server_base.cc:1034] running on GCE node
I20260928 23:52:14.890641 20522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:14.891724 20522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:14.892916 20522 hybrid_clock.cc:648] HybridClock initialized: now 1790639534892896 us; error 32 us; skew 500 ppm
Sep 28 23:52:14 dist-test-slave-p5bw krb5kdc[20509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639534, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:14.903374 20522 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:52:14.904379 20522 webserver.cc:460] Webserver started at http://127.6.152.62:33947/ using document root <none> and password file <none>
I20260928 23:52:14.905036 20522 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:14.905102 20522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:14.905356 20522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:52:14.907400 20522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "d04ec9210a964749a041cf9a3d387ae3"
format_stamp: "Formatted at 2026-09-28 23:52:14 on dist-test-slave-p5bw"
server_key: "1885f4b76284d2205900a43d2567952b"
server_key_iv: "2ea3b37a987ea551b3eb93a9238158dd"
server_key_version: "encryptionkey@0"
I20260928 23:52:14.907951 20522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "d04ec9210a964749a041cf9a3d387ae3"
format_stamp: "Formatted at 2026-09-28 23:52:14 on dist-test-slave-p5bw"
server_key: "1885f4b76284d2205900a43d2567952b"
server_key_iv: "2ea3b37a987ea551b3eb93a9238158dd"
server_key_version: "encryptionkey@0"
I20260928 23:52:14.911981 20522 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:52:14.914531 20537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:14.915947 20522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:52:14.916109 20522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "d04ec9210a964749a041cf9a3d387ae3"
format_stamp: "Formatted at 2026-09-28 23:52:14 on dist-test-slave-p5bw"
server_key: "1885f4b76284d2205900a43d2567952b"
server_key_iv: "2ea3b37a987ea551b3eb93a9238158dd"
server_key_version: "encryptionkey@0"
I20260928 23:52:14.916250 20522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:14.937122 20522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:14.940279 20522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:14.940496 20522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:14.947388 20522 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:38395
I20260928 23:52:14.947672 20589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:38395 every 8 connection(s)
I20260928 23:52:14.948414 20522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:52:14.951422 20590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:52:14.956099 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 20522
I20260928 23:52:14.956267 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:52:14.956540 6752 external_mini_cluster.cc:1428] Setting key 32afde9d48aef80a732a8e170f4dbf01
I20260928 23:52:14.957192 20590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3: Bootstrap starting.
I20260928 23:52:14.959960 20590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3: Neither blocks nor log segments found. Creating new log.
I20260928 23:52:14.960811 20590 log.cc:824] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:14.963060 20590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3: No bootstrap required, opened a new log
Sep 28 23:52:14 dist-test-slave-p5bw krb5kdc[20509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:52:14.967307 20590 raft_consensus.cc:348] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04ec9210a964749a041cf9a3d387ae3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38395 } }
I20260928 23:52:14.967634 20590 raft_consensus.cc:374] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:52:14.967716 20590 raft_consensus.cc:729] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d04ec9210a964749a041cf9a3d387ae3, State: Initialized, Role: FOLLOWER
I20260928 23:52:14.968286 20590 consensus_queue.cc:260] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04ec9210a964749a041cf9a3d387ae3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38395 } }
I20260928 23:52:14.968472 20590 raft_consensus.cc:388] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:52:14.968559 20590 raft_consensus.cc:482] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:52:14.968672 20590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:52:14.969928 20590 raft_consensus.cc:504] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04ec9210a964749a041cf9a3d387ae3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38395 } }
I20260928 23:52:14.970311 20590 leader_election.cc:302] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d04ec9210a964749a041cf9a3d387ae3; no voters:
I20260928 23:52:14.970713 20590 leader_election.cc:288] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:52:14.970918 20595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:52:14.971138 20595 raft_consensus.cc:686] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [term 1 LEADER]: Becoming Leader. State: Replica: d04ec9210a964749a041cf9a3d387ae3, State: Running, Role: LEADER
I20260928 23:52:14.971536 20595 consensus_queue.cc:237] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04ec9210a964749a041cf9a3d387ae3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38395 } }
I20260928 23:52:14.972069 20590 sys_catalog.cc:564] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:52:14.972863 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:14.957930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44594 (local address 127.6.152.62:38395)
0928 23:52:14.958417 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:14.958438 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:14.958486 (+ 48us) server_negotiation.cc:407] Connection header received
0928 23:52:14.959294 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:14.959319 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:14.959733 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:14.960097 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:14.960870 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:14.962446 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:14.963435 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:14.964144 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:14.964308 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:14.967135 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:14.967141 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:14.967158 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:14.967190 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:14.969430 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:14.969888 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:14.969896 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:14.969904 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:14.970080 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:14.970417 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:14.970422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:14.970427 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:14.970977 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:14.971184 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:14.971533 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 23:52:14.971861 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":159,"threads_started":1}
I20260928 23:52:14.973201 20596 sys_catalog.cc:455] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d04ec9210a964749a041cf9a3d387ae3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04ec9210a964749a041cf9a3d387ae3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38395 } } }
I20260928 23:52:14.973330 20596 sys_catalog.cc:458] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [sys.catalog]: This master's current role is: LEADER
I20260928 23:52:14.973610 20597 sys_catalog.cc:455] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d04ec9210a964749a041cf9a3d387ae3. Latest consensus state: current_term: 1 leader_uuid: "d04ec9210a964749a041cf9a3d387ae3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04ec9210a964749a041cf9a3d387ae3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38395 } } }
I20260928 23:52:14.973693 20597 sys_catalog.cc:458] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [sys.catalog]: This master's current role is: LEADER
W20260928 23:52:14.977283 20610 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:52:14.977351 20610 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:52:14.977429 20604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:52:14.978188 20604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:52:14.984112 20604 catalog_manager.cc:1347] Generated new cluster ID: 3ab7fb17a3d446bdb33ee47962501e5d
I20260928 23:52:14.984177 20604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:52:15.000882 20604 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:52:15.002182 20604 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:52:15.007555 20604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3: Generated new TSK 0
I20260928 23:52:15.008378 20604 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
I20260928 23:52:15.045634 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:38395
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:52:15.155627 20616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:15.155936 20616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:15.156010 20616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:15.160112 20616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:15.160205 20616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:15.160300 20616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:52:15.165587 20616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:15.166671 20616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:15.167689 20616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:15.177855 20621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:15.177919 20622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:15.177894 20616 server_base.cc:1034] running on GCE node
W20260928 23:52:15.177858 20624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:15.178509 20616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:15.179193 20616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:15.180356 20616 hybrid_clock.cc:648] HybridClock initialized: now 1790639535180341 us; error 27 us; skew 500 ppm
Sep 28 23:52:15 dist-test-slave-p5bw krb5kdc[20509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639535, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:15.183091 20616 init.cc:377] Logged in from keytab as kudu/127.6.152.1@KRBTEST.COM (short username kudu)
I20260928 23:52:15.183910 20616 webserver.cc:460] Webserver started at http://127.6.152.1:37833/ using document root <none> and password file <none>
I20260928 23:52:15.184413 20616 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:15.184471 20616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:15.184631 20616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:52:15.186597 20616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "3ddfefce2d894f6697bc7011773750bf"
format_stamp: "Formatted at 2026-09-28 23:52:15 on dist-test-slave-p5bw"
server_key: "3aedbeb410904fbcba587d29f530446f"
server_key_iv: "28fe90867f765bd5a57dfdcafa757337"
server_key_version: "encryptionkey@0"
I20260928 23:52:15.187083 20616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "3ddfefce2d894f6697bc7011773750bf"
format_stamp: "Formatted at 2026-09-28 23:52:15 on dist-test-slave-p5bw"
server_key: "3aedbeb410904fbcba587d29f530446f"
server_key_iv: "28fe90867f765bd5a57dfdcafa757337"
server_key_version: "encryptionkey@0"
I20260928 23:52:15.190613 20616 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 23:52:15.192938 20631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:15.194088 20616 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:52:15.194259 20616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "3ddfefce2d894f6697bc7011773750bf"
format_stamp: "Formatted at 2026-09-28 23:52:15 on dist-test-slave-p5bw"
server_key: "3aedbeb410904fbcba587d29f530446f"
server_key_iv: "28fe90867f765bd5a57dfdcafa757337"
server_key_version: "encryptionkey@0"
I20260928 23:52:15.194372 20616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:15.206449 20616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:15.209216 20616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:15.209407 20616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:15.209955 20616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:52:15.210727 20616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:52:15.210796 20616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:15.210868 20616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:52:15.210913 20616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:15.219425 20616 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:36277
I20260928 23:52:15.219451 20744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:36277 every 8 connection(s)
I20260928 23:52:15.220394 20616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:52:15.221879 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 20616
I20260928 23:52:15.222014 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
I20260928 23:52:15.222337 6752 external_mini_cluster.cc:1428] Setting key 10c7949e3aba659690725703df1a6e45
Sep 28 23:52:15 dist-test-slave-p5bw krb5kdc[20509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639535, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.152.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.2@KRBTEST.COM" created.
I20260928 23:52:15.234393 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:15.222244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:42601 (local address 127.6.152.62:38395)
0928 23:52:15.222393 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:15.222400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:15.223945 (+ 1545us) server_negotiation.cc:407] Connection header received
0928 23:52:15.224851 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:15.224855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:15.224908 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:15.224982 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:15.226115 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.226643 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.227796 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.228031 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.228139 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:15.230613 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:15.230616 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:15.230618 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:15.230640 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:15.232205 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:15.232720 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:15.232724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:15.232726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:15.232869 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:15.233238 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:15.233242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:15.233244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:15.233498 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:15.233610 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:15.234083 (+ 473us) server_negotiation.cc:299] Negotiation successful
0928 23:52:15.234218 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:52:15.235198 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:15.222744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.1:42601)
0928 23:52:15.223792 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:15.223853 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:15.224628 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:15.225150 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:15.225157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:15.225510 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:15.225967 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:15.225980 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.226776 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.226780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:15.227657 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:15.227666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.228181 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.228185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:15.228359 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:15.228993 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:15.229008 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:15.230443 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:15.232332 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:15.232337 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:15.232351 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:15.232601 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:15.232990 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:15.232993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:15.232995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:15.233139 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:15.233608 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:15.233615 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:15.233857 (+ 242us) client_negotiation.cc:769] Sending connection context
0928 23:52:15.234042 (+ 185us) client_negotiation.cc:240] Negotiation successful
0928 23:52:15.234356 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":661,"threads_started":1}
I20260928 23:52:15.236424 20745 heartbeater.cc:347] Connected to a master server at 127.6.152.62:38395
I20260928 23:52:15.236670 20745 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:15.237195 20745 heartbeater.cc:511] Master 127.6.152.62:38395 requested a full tablet report, sending...
I20260928 23:52:15.238858 20554 ts_manager.cc:194] Registered new tserver with Master: 3ddfefce2d894f6697bc7011773750bf (127.6.152.1:36277)
I20260928 23:52:15.240512 20554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.1@KRBTEST.COM'} at 127.6.152.1:42601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
I20260928 23:52:15.249140 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:38395
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:52:15.361836 20750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:15.362104 20750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:15.362149 20750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:15.366487 20750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:15.366549 20750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:15.366609 20750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:52:15.371757 20750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:15.372962 20750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:15.374106 20750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:15.383600 20756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:15.383576 20755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:15.383677 20758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:15.383797 20750 server_base.cc:1034] running on GCE node
I20260928 23:52:15.384214 20750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:15.384874 20750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:15.386039 20750 hybrid_clock.cc:648] HybridClock initialized: now 1790639535386020 us; error 29 us; skew 500 ppm
Sep 28 23:52:15 dist-test-slave-p5bw krb5kdc[20509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639535, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:15.388868 20750 init.cc:377] Logged in from keytab as kudu/127.6.152.2@KRBTEST.COM (short username kudu)
I20260928 23:52:15.389741 20750 webserver.cc:460] Webserver started at http://127.6.152.2:35505/ using document root <none> and password file <none>
I20260928 23:52:15.390290 20750 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:15.390348 20750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:15.390568 20750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:52:15.392593 20750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "5800969806274705b9f893edd0f6a069"
format_stamp: "Formatted at 2026-09-28 23:52:15 on dist-test-slave-p5bw"
server_key: "1f45295d76092946fe0cd5ea6b7a8a9d"
server_key_iv: "b444aa4e08a90c67bcf9b8b1ac81b01a"
server_key_version: "encryptionkey@0"
I20260928 23:52:15.393186 20750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "5800969806274705b9f893edd0f6a069"
format_stamp: "Formatted at 2026-09-28 23:52:15 on dist-test-slave-p5bw"
server_key: "1f45295d76092946fe0cd5ea6b7a8a9d"
server_key_iv: "b444aa4e08a90c67bcf9b8b1ac81b01a"
server_key_version: "encryptionkey@0"
I20260928 23:52:15.396770 20750 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:52:15.399406 20765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:15.400630 20750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:52:15.400761 20750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "5800969806274705b9f893edd0f6a069"
format_stamp: "Formatted at 2026-09-28 23:52:15 on dist-test-slave-p5bw"
server_key: "1f45295d76092946fe0cd5ea6b7a8a9d"
server_key_iv: "b444aa4e08a90c67bcf9b8b1ac81b01a"
server_key_version: "encryptionkey@0"
I20260928 23:52:15.400918 20750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:15.415683 20750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:15.418843 20750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:15.419034 20750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:15.419628 20750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:52:15.420531 20750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:52:15.420575 20750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:15.420641 20750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:52:15.420673 20750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:15.429989 20750 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:43285
I20260928 23:52:15.430007 20878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:43285 every 8 connection(s)
I20260928 23:52:15.430928 20750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:52:15.435122 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 20750
I20260928 23:52:15.435284 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:52:15.435570 6752 external_mini_cluster.cc:1428] Setting key 356f03775c23036cd426ffc04150a0b7
Sep 28 23:52:15 dist-test-slave-p5bw krb5kdc[20509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639535, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:52:15.445524 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:15.432736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:35089 (local address 127.6.152.62:38395)
0928 23:52:15.432901 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:15.432905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:15.433661 (+ 756us) server_negotiation.cc:407] Connection header received
0928 23:52:15.434937 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:15.434941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:15.434998 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:15.435073 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:15.436501 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.437254 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.438381 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.438753 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.438840 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:15.441630 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:15.441633 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:15.441636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:15.441658 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:15.443411 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:15.443929 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:15.443935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:15.443936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:15.444027 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:15.444351 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:15.444356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:15.444358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:15.444583 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:15.444694 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:15.445199 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 23:52:15.445336 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.6.152.3@KRBTEST.COM; defaulting to no policy
I20260928 23:52:15.446249 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:15.433081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.2:35089)
0928 23:52:15.433504 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:15.433546 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:15.434723 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:15.435272 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:15.435282 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:15.435733 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:15.436352 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:15.436372 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.437381 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.437384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:15.438250 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:15.438259 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.438921 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.438928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:15.439143 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:15.439772 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:15.439791 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:15.441461 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:15.443536 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:15.443540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:15.443551 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:15.443824 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:15.444127 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:15.444129 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:15.444130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:15.444254 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:15.444689 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:15.444693 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:15.444962 (+ 269us) client_negotiation.cc:769] Sending connection context
0928 23:52:15.445124 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 23:52:15.445358 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":110,"threads_started":1}
I20260928 23:52:15.447299 20879 heartbeater.cc:347] Connected to a master server at 127.6.152.62:38395
I20260928 23:52:15.447494 20879 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:15.447898 20879 heartbeater.cc:511] Master 127.6.152.62:38395 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.3@KRBTEST.COM" created.
I20260928 23:52:15.448905 20554 ts_manager.cc:194] Registered new tserver with Master: 5800969806274705b9f893edd0f6a069 (127.6.152.2:43285)
I20260928 23:52:15.449462 20554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.2@KRBTEST.COM'} at 127.6.152.2:35089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
I20260928 23:52:15.461915 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:38395
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:52:15.570667 20884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:15.570955 20884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:15.571028 20884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:15.575784 20884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:15.575858 20884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:15.575954 20884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:52:15.581700 20884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:15.582705 20884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:15.583700 20884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:15.592620 20890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:15.592640 20889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:15.592679 20884 server_base.cc:1034] running on GCE node
W20260928 23:52:15.592656 20892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:15.593278 20884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:15.593886 20884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:15.595036 20884 hybrid_clock.cc:648] HybridClock initialized: now 1790639535595024 us; error 24 us; skew 500 ppm
Sep 28 23:52:15 dist-test-slave-p5bw krb5kdc[20509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639535, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:15.597702 20884 init.cc:377] Logged in from keytab as kudu/127.6.152.3@KRBTEST.COM (short username kudu)
I20260928 23:52:15.598501 20884 webserver.cc:460] Webserver started at http://127.6.152.3:35319/ using document root <none> and password file <none>
I20260928 23:52:15.599056 20884 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:15.599114 20884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:15.599272 20884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:52:15.601279 20884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "ed0f7ba578aa4ba2b5132c9f4741190f"
format_stamp: "Formatted at 2026-09-28 23:52:15 on dist-test-slave-p5bw"
server_key: "4e0d9997cbb1330a432433630f406c41"
server_key_iv: "6eb8620012a4a37648682c2e55361d3f"
server_key_version: "encryptionkey@0"
I20260928 23:52:15.601768 20884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "ed0f7ba578aa4ba2b5132c9f4741190f"
format_stamp: "Formatted at 2026-09-28 23:52:15 on dist-test-slave-p5bw"
server_key: "4e0d9997cbb1330a432433630f406c41"
server_key_iv: "6eb8620012a4a37648682c2e55361d3f"
server_key_version: "encryptionkey@0"
I20260928 23:52:15.605458 20884 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:52:15.607828 20899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:15.609076 20884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:52:15.609200 20884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "ed0f7ba578aa4ba2b5132c9f4741190f"
format_stamp: "Formatted at 2026-09-28 23:52:15 on dist-test-slave-p5bw"
server_key: "4e0d9997cbb1330a432433630f406c41"
server_key_iv: "6eb8620012a4a37648682c2e55361d3f"
server_key_version: "encryptionkey@0"
I20260928 23:52:15.609305 20884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:15.634620 20884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:15.637683 20884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:15.637892 20884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:15.638468 20884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:52:15.639343 20884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:52:15.639410 20884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:15.639482 20884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:52:15.639533 20884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:15.648053 20884 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:43391
I20260928 23:52:15.648075 21012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:43391 every 8 connection(s)
I20260928 23:52:15.649055 20884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:52:15.657624 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 20884
I20260928 23:52:15.657755 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:52:15.658061 6752 external_mini_cluster.cc:1428] Setting key 6427b3bde19b1920690e1949256a466b
Sep 28 23:52:15 dist-test-slave-p5bw krb5kdc[20509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639535, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:52:15.663240 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:15.650748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:54721 (local address 127.6.152.62:38395)
0928 23:52:15.650893 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:15.650897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:15.651700 (+ 803us) server_negotiation.cc:407] Connection header received
0928 23:52:15.652995 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:15.652998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:15.653046 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:15.653120 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:15.654429 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.655373 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.656777 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.657170 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.657300 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:15.660173 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:15.660176 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:15.660180 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:15.660208 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:15.661519 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:15.661967 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:15.661969 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:15.661971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:15.662042 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:15.662329 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:15.662332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:15.662333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:15.662493 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:15.662566 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:15.663011 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 23:52:15.663121 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 23:52:15.664048 21015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:15.651119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.3:54721)
0928 23:52:15.651539 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:15.651582 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:15.652723 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:15.653272 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:15.653278 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:15.653619 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:15.654244 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:15.654274 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.655516 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.655519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:15.656650 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:15.656659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.657331 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.657335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:15.657545 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:15.658137 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:15.658158 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:15.660001 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:15.661616 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:15.661621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:15.661633 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:15.661885 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:15.662117 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:15.662119 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:15.662120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:15.662253 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:15.662576 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:15.662581 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:15.662816 (+ 235us) client_negotiation.cc:769] Sending connection context
0928 23:52:15.662965 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 23:52:15.663227 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":114,"threads_started":1}
I20260928 23:52:15.665158 21013 heartbeater.cc:347] Connected to a master server at 127.6.152.62:38395
I20260928 23:52:15.665409 21013 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:15.665870 21013 heartbeater.cc:511] Master 127.6.152.62:38395 requested a full tablet report, sending...
I20260928 23:52:15.666841 20554 ts_manager.cc:194] Registered new tserver with Master: ed0f7ba578aa4ba2b5132c9f4741190f (127.6.152.3:43391)
I20260928 23:52:15.667395 20554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.3@KRBTEST.COM'} at 127.6.152.3:54721
I20260928 23:52:15.671882 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:52:15.681035 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:15.673896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44610 (local address 127.6.152.62:38395)
0928 23:52:15.674061 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:15.674065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:15.674296 (+ 231us) server_negotiation.cc:407] Connection header received
0928 23:52:15.674352 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:15.674354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:15.674404 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:15.674481 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:15.675089 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.675578 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.676387 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.676621 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.676709 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:15.678021 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:15.678024 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:15.678026 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:15.678047 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:15.679346 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:15.679880 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:15.679883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:15.679884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:15.679981 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:15.680225 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:15.680227 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:15.680228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:15.680422 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:15.680520 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:15.680699 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 23:52:15.680796 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 23:52:15.683851 20554 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:52:15.686868 20554 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:52:15.700886 21025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:15.694842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:36277 (local address 127.0.0.1:44886)
0928 23:52:15.695707 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:15.695759 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:15.695899 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:15.696755 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:15.696758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:15.696777 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:52:15.696898 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:15.696904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.698035 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.698041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:15.699347 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:15.699356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.700295 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.700299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:15.700408 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:15.700457 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 23:52:15.700567 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 23:52:15.700694 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":31,"thread_start_us":81,"threads_started":1}
I20260928 23:52:15.700905 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:15.695364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44886 (local address 127.6.152.1:36277)
0928 23:52:15.696363 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:15.696377 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:15.696397 (+ 20us) server_negotiation.cc:407] Connection header received
0928 23:52:15.696461 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:15.696465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:15.696614 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:15.696747 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:52:15.697011 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.697919 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.699468 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.700159 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.700266 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:15.700416 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:15.700617 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 23:52:15.700729 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898,"thread_start_us":68,"threads_started":1}
I20260928 23:52:15.701287 21026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:15.694887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34364 (local address 127.6.152.3:43391)
0928 23:52:15.695737 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:15.695750 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:15.696159 (+ 409us) server_negotiation.cc:407] Connection header received
0928 23:52:15.696296 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:15.696302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:15.696479 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:15.696614 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:52:15.697113 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.698193 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.699760 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.700417 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.700495 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:15.700636 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:15.701013 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 23:52:15.701133 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":81,"threads_started":1}
I20260928 23:52:15.701447 21024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:15.695101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:43285 (local address 127.0.0.1:41676)
0928 23:52:15.695870 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:15.695882 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:15.695947 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:15.696472 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:15.696477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:15.696498 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:52:15.696639 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:15.696651 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.698321 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.698325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:15.699744 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:15.699752 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.700794 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.700798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:15.701162 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:15.701186 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 23:52:15.701248 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 23:52:15.701314 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":232,"thread_start_us":97,"threads_started":1}
I20260928 23:52:15.701707 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:15.694773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:43391 (local address 127.0.0.1:34364)
0928 23:52:15.696053 (+ 1280us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:15.696068 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:15.696157 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:15.696842 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:15.696845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:15.696863 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:52:15.696988 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:15.696996 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.698326 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.698328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:15.699612 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:15.699622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.700735 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.700738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:15.700854 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:15.700876 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:52:15.701516 (+ 640us) client_negotiation.cc:240] Negotiation successful
0928 23:52:15.701593 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":26,"thread_start_us":81,"threads_started":1}
I20260928 23:52:15.702121 21029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:15.695538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41676 (local address 127.6.152.2:43285)
0928 23:52:15.695990 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:15.696006 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:15.696029 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:52:15.696109 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:15.696117 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:15.696327 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:15.696483 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:52:15.696791 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.698156 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.699857 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.700651 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.701686 (+ 1035us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:15.701841 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:15.701891 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 23:52:15.701984 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":105,"threads_started":1}
I20260928 23:52:15.704216 20947 tablet_service.cc:1467] Processing CreateTablet for tablet 0eb643f059634de3938516f91d5c6fb9 (DEFAULT_TABLE table=test-table [id=64c3f5d6857a442aa7f64bad2f68fdd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:52:15.704248 20679 tablet_service.cc:1467] Processing CreateTablet for tablet 0eb643f059634de3938516f91d5c6fb9 (DEFAULT_TABLE table=test-table [id=64c3f5d6857a442aa7f64bad2f68fdd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:52:15.704493 20813 tablet_service.cc:1467] Processing CreateTablet for tablet 0eb643f059634de3938516f91d5c6fb9 (DEFAULT_TABLE table=test-table [id=64c3f5d6857a442aa7f64bad2f68fdd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:52:15.705265 20679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eb643f059634de3938516f91d5c6fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:52:15.705265 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eb643f059634de3938516f91d5c6fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:52:15.705327 20813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eb643f059634de3938516f91d5c6fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:52:15.711166 21030 tablet_bootstrap.cc:492] T 0eb643f059634de3938516f91d5c6fb9 P ed0f7ba578aa4ba2b5132c9f4741190f: Bootstrap starting.
I20260928 23:52:15.712594 21031 tablet_bootstrap.cc:492] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf: Bootstrap starting.
I20260928 23:52:15.712955 21032 tablet_bootstrap.cc:492] T 0eb643f059634de3938516f91d5c6fb9 P 5800969806274705b9f893edd0f6a069: Bootstrap starting.
I20260928 23:52:15.713727 21030 tablet_bootstrap.cc:654] T 0eb643f059634de3938516f91d5c6fb9 P ed0f7ba578aa4ba2b5132c9f4741190f: Neither blocks nor log segments found. Creating new log.
I20260928 23:52:15.714461 21030 log.cc:824] T 0eb643f059634de3938516f91d5c6fb9 P ed0f7ba578aa4ba2b5132c9f4741190f: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:15.714818 21031 tablet_bootstrap.cc:654] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf: Neither blocks nor log segments found. Creating new log.
I20260928 23:52:15.715247 21032 tablet_bootstrap.cc:654] T 0eb643f059634de3938516f91d5c6fb9 P 5800969806274705b9f893edd0f6a069: Neither blocks nor log segments found. Creating new log.
I20260928 23:52:15.716049 21032 log.cc:824] T 0eb643f059634de3938516f91d5c6fb9 P 5800969806274705b9f893edd0f6a069: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:15.716048 21031 log.cc:824] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:15.716918 21030 tablet_bootstrap.cc:492] T 0eb643f059634de3938516f91d5c6fb9 P ed0f7ba578aa4ba2b5132c9f4741190f: No bootstrap required, opened a new log
I20260928 23:52:15.717109 21030 ts_tablet_manager.cc:1397] T 0eb643f059634de3938516f91d5c6fb9 P ed0f7ba578aa4ba2b5132c9f4741190f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:52:15.717954 21031 tablet_bootstrap.cc:492] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf: No bootstrap required, opened a new log
I20260928 23:52:15.718125 21031 ts_tablet_manager.cc:1397] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:52:15.718245 21032 tablet_bootstrap.cc:492] T 0eb643f059634de3938516f91d5c6fb9 P 5800969806274705b9f893edd0f6a069: No bootstrap required, opened a new log
I20260928 23:52:15.718410 21032 ts_tablet_manager.cc:1397] T 0eb643f059634de3938516f91d5c6fb9 P 5800969806274705b9f893edd0f6a069: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:52:15.720227 21030 raft_consensus.cc:348] T 0eb643f059634de3938516f91d5c6fb9 P ed0f7ba578aa4ba2b5132c9f4741190f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } }
I20260928 23:52:15.720510 21030 raft_consensus.cc:374] T 0eb643f059634de3938516f91d5c6fb9 P ed0f7ba578aa4ba2b5132c9f4741190f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:52:15.720563 21030 raft_consensus.cc:729] T 0eb643f059634de3938516f91d5c6fb9 P ed0f7ba578aa4ba2b5132c9f4741190f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed0f7ba578aa4ba2b5132c9f4741190f, State: Initialized, Role: FOLLOWER
I20260928 23:52:15.721042 21030 consensus_queue.cc:260] T 0eb643f059634de3938516f91d5c6fb9 P ed0f7ba578aa4ba2b5132c9f4741190f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } }
I20260928 23:52:15.721660 21013 heartbeater.cc:503] Master 127.6.152.62:38395 was elected leader, sending a full tablet report...
I20260928 23:52:15.721835 21031 raft_consensus.cc:348] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } }
I20260928 23:52:15.722059 21031 raft_consensus.cc:374] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:52:15.722110 21031 raft_consensus.cc:729] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ddfefce2d894f6697bc7011773750bf, State: Initialized, Role: FOLLOWER
I20260928 23:52:15.722128 21030 ts_tablet_manager.cc:1428] T 0eb643f059634de3938516f91d5c6fb9 P ed0f7ba578aa4ba2b5132c9f4741190f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 23:52:15.722532 21031 consensus_queue.cc:260] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } }
I20260928 23:52:15.723129 21032 raft_consensus.cc:348] T 0eb643f059634de3938516f91d5c6fb9 P 5800969806274705b9f893edd0f6a069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } }
I20260928 23:52:15.723345 21032 raft_consensus.cc:374] T 0eb643f059634de3938516f91d5c6fb9 P 5800969806274705b9f893edd0f6a069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:52:15.723404 21032 raft_consensus.cc:729] T 0eb643f059634de3938516f91d5c6fb9 P 5800969806274705b9f893edd0f6a069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5800969806274705b9f893edd0f6a069, State: Initialized, Role: FOLLOWER
I20260928 23:52:15.723456 21031 ts_tablet_manager.cc:1428] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 23:52:15.723764 21032 consensus_queue.cc:260] T 0eb643f059634de3938516f91d5c6fb9 P 5800969806274705b9f893edd0f6a069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } }
I20260928 23:52:15.724428 20879 heartbeater.cc:503] Master 127.6.152.62:38395 was elected leader, sending a full tablet report...
I20260928 23:52:15.724530 21032 ts_tablet_manager.cc:1428] T 0eb643f059634de3938516f91d5c6fb9 P 5800969806274705b9f893edd0f6a069: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:52:15.724553 20745 heartbeater.cc:503] Master 127.6.152.62:38395 was elected leader, sending a full tablet report...
I20260928 23:52:15.829057 21037 raft_consensus.cc:482] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:52:15.829335 21037 raft_consensus.cc:504] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } }
I20260928 23:52:15.830464 21037 leader_election.cc:288] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed0f7ba578aa4ba2b5132c9f4741190f (127.6.152.3:43391), 5800969806274705b9f893edd0f6a069 (127.6.152.2:43285)
I20260928 23:52:15.835865 21029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:15.830871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:58041 (local address 127.6.152.2:43285)
0928 23:52:15.830995 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:15.831000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:15.831218 (+ 218us) server_negotiation.cc:407] Connection header received
0928 23:52:15.831385 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:15.831388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:15.831441 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:15.831537 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:52:15.831947 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.832722 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.834584 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.835185 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.835290 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:15.835351 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:15.835675 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 23:52:15.835740 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:52:15.835865 21026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:15.830871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:36789 (local address 127.6.152.3:43391)
0928 23:52:15.830996 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:15.831000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:15.831335 (+ 335us) server_negotiation.cc:407] Connection header received
0928 23:52:15.831390 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:15.831393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:15.831441 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:15.831536 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:52:15.831947 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.832722 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.834584 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.835185 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.835296 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:15.835351 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:15.835675 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 23:52:15.835739 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:52:15.835923 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:15.830744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:43391 (local address 127.6.152.1:36789)
0928 23:52:15.831207 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:15.831219 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:15.831321 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:15.831565 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:15.831568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:15.831609 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:52:15.831835 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:15.831842 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.832857 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.832860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:15.834455 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:15.834466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.835311 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.835315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:15.835482 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:15.835564 (+ 82us) client_negotiation.cc:769] Sending connection context
0928 23:52:15.835643 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 23:52:15.835741 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":250,"spinlock_wait_cycles":7168,"thread_start_us":94,"threads_started":1}
I20260928 23:52:15.835925 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:15.830737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:43285 (local address 127.6.152.1:58041)
0928 23:52:15.831118 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:15.831134 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:15.831261 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:15.831565 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:15.831568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:15.831607 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:52:15.831824 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:15.831833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.832857 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.832860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:15.834463 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:15.834470 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.835312 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.835315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:15.835508 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:15.835570 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 23:52:15.835643 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 23:52:15.835741 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":398,"spinlock_wait_cycles":29696,"thread_start_us":89,"threads_started":1}
I20260928 23:52:15.836771 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb643f059634de3938516f91d5c6fb9" candidate_uuid: "3ddfefce2d894f6697bc7011773750bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" is_pre_election: true
I20260928 23:52:15.836771 20833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb643f059634de3938516f91d5c6fb9" candidate_uuid: "3ddfefce2d894f6697bc7011773750bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5800969806274705b9f893edd0f6a069" is_pre_election: true
I20260928 23:52:15.837106 20833 raft_consensus.cc:2454] T 0eb643f059634de3938516f91d5c6fb9 P 5800969806274705b9f893edd0f6a069 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ddfefce2d894f6697bc7011773750bf in term 0.
I20260928 23:52:15.837106 20967 raft_consensus.cc:2454] T 0eb643f059634de3938516f91d5c6fb9 P ed0f7ba578aa4ba2b5132c9f4741190f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ddfefce2d894f6697bc7011773750bf in term 0.
I20260928 23:52:15.837628 20635 leader_election.cc:302] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3ddfefce2d894f6697bc7011773750bf, 5800969806274705b9f893edd0f6a069, ed0f7ba578aa4ba2b5132c9f4741190f; no voters:
I20260928 23:52:15.837936 21037 raft_consensus.cc:2781] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:52:15.838032 21037 raft_consensus.cc:482] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:52:15.838114 21037 raft_consensus.cc:3037] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:52:15.839152 21037 raft_consensus.cc:504] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } }
I20260928 23:52:15.839490 21037 leader_election.cc:288] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 1 election: Requested vote from peers ed0f7ba578aa4ba2b5132c9f4741190f (127.6.152.3:43391), 5800969806274705b9f893edd0f6a069 (127.6.152.2:43285)
I20260928 23:52:15.839854 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb643f059634de3938516f91d5c6fb9" candidate_uuid: "3ddfefce2d894f6697bc7011773750bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f"
I20260928 23:52:15.839854 20833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb643f059634de3938516f91d5c6fb9" candidate_uuid: "3ddfefce2d894f6697bc7011773750bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5800969806274705b9f893edd0f6a069"
I20260928 23:52:15.839972 20833 raft_consensus.cc:3037] T 0eb643f059634de3938516f91d5c6fb9 P 5800969806274705b9f893edd0f6a069 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:52:15.839972 20967 raft_consensus.cc:3037] T 0eb643f059634de3938516f91d5c6fb9 P ed0f7ba578aa4ba2b5132c9f4741190f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:52:15.841231 20967 raft_consensus.cc:2454] T 0eb643f059634de3938516f91d5c6fb9 P ed0f7ba578aa4ba2b5132c9f4741190f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ddfefce2d894f6697bc7011773750bf in term 1.
I20260928 23:52:15.841231 20833 raft_consensus.cc:2454] T 0eb643f059634de3938516f91d5c6fb9 P 5800969806274705b9f893edd0f6a069 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ddfefce2d894f6697bc7011773750bf in term 1.
I20260928 23:52:15.841537 20635 leader_election.cc:302] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ddfefce2d894f6697bc7011773750bf, 5800969806274705b9f893edd0f6a069; no voters:
I20260928 23:52:15.841769 21037 raft_consensus.cc:2781] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:52:15.842029 21037 raft_consensus.cc:686] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf [term 1 LEADER]: Becoming Leader. State: Replica: 3ddfefce2d894f6697bc7011773750bf, State: Running, Role: LEADER
I20260928 23:52:15.842355 21037 consensus_queue.cc:237] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } }
I20260928 23:52:15.845479 20554 catalog_manager.cc:5495] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ddfefce2d894f6697bc7011773750bf (127.6.152.1). New cstate: current_term: 1 leader_uuid: "3ddfefce2d894f6697bc7011773750bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } health_report { overall_health: UNKNOWN } } }
I20260928 23:52:15.887418 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:15.884115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44888 (local address 127.6.152.1:36277)
0928 23:52:15.884233 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:15.884236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:15.884249 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:52:15.884350 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:15.884353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:15.884409 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:15.884500 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:52:15.884738 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.885264 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.886084 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:15.886289 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:15.886358 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:15.886604 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:52:15.887021 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:52:15.887109 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:15.887181 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 23:52:15.887251 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:52:15.893256 20833 raft_consensus.cc:1261] T 0eb643f059634de3938516f91d5c6fb9 P 5800969806274705b9f893edd0f6a069 [term 1 FOLLOWER]: Refusing update from remote peer 3ddfefce2d894f6697bc7011773750bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:52:15.893256 20967 raft_consensus.cc:1261] T 0eb643f059634de3938516f91d5c6fb9 P ed0f7ba578aa4ba2b5132c9f4741190f [term 1 FOLLOWER]: Refusing update from remote peer 3ddfefce2d894f6697bc7011773750bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:52:15.893982 21041 consensus_queue.cc:1035] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:52:15.894217 21037 consensus_queue.cc:1035] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 23:52:15.899915 21014 tablet.cc:2369] T 0eb643f059634de3938516f91d5c6fb9 P ed0f7ba578aa4ba2b5132c9f4741190f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:52:15.903954 21045 mvcc.cc:205] Tried to move back new op lower bound from 7334459539011706880 to 7334459538813468672. Current Snapshot: MvccSnapshot[applied={T|T < 7334459539011706880 or (T in {7334459539011706880})}]
I20260928 23:52:15.909669 20554 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:52:15.909844 20554 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:52:15.912248 20554 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3: Sending DeleteTablet for 3 replicas of tablet 0eb643f059634de3938516f91d5c6fb9
I20260928 23:52:15.913012 20947 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eb643f059634de3938516f91d5c6fb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:52:15 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:34364
I20260928 23:52:15.913013 20679 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eb643f059634de3938516f91d5c6fb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:52:15 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:44886
I20260928 23:52:15.913079 20813 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eb643f059634de3938516f91d5c6fb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:52:15 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:41676
I20260928 23:52:15.913448 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 20616
I20260928 23:52:15.914192 21055 tablet_replica.cc:330] T 0eb643f059634de3938516f91d5c6fb9 P 5800969806274705b9f893edd0f6a069: stopping tablet replica
I20260928 23:52:15.914489 21055 raft_consensus.cc:2229] T 0eb643f059634de3938516f91d5c6fb9 P 5800969806274705b9f893edd0f6a069 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:52:15.914759 21053 tablet_replica.cc:330] T 0eb643f059634de3938516f91d5c6fb9 P ed0f7ba578aa4ba2b5132c9f4741190f: stopping tablet replica
I20260928 23:52:15.914791 21055 raft_consensus.cc:2258] T 0eb643f059634de3938516f91d5c6fb9 P 5800969806274705b9f893edd0f6a069 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:52:15.915067 21053 raft_consensus.cc:2229] T 0eb643f059634de3938516f91d5c6fb9 P ed0f7ba578aa4ba2b5132c9f4741190f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:52:15.915359 21053 raft_consensus.cc:2258] T 0eb643f059634de3938516f91d5c6fb9 P ed0f7ba578aa4ba2b5132c9f4741190f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:52:15.916138 21055 ts_tablet_manager.cc:1905] T 0eb643f059634de3938516f91d5c6fb9 P 5800969806274705b9f893edd0f6a069: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:52:15.916842 21053 ts_tablet_manager.cc:1905] T 0eb643f059634de3938516f91d5c6fb9 P ed0f7ba578aa4ba2b5132c9f4741190f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:52:15.918361 21055 ts_tablet_manager.cc:1918] T 0eb643f059634de3938516f91d5c6fb9 P 5800969806274705b9f893edd0f6a069: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:52:15.918470 21055 log.cc:1196] T 0eb643f059634de3938516f91d5c6fb9 P 5800969806274705b9f893edd0f6a069: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/wal/wals/0eb643f059634de3938516f91d5c6fb9
I20260928 23:52:15.918748 21055 ts_tablet_manager.cc:1939] T 0eb643f059634de3938516f91d5c6fb9 P 5800969806274705b9f893edd0f6a069: Deleting consensus metadata
I20260928 23:52:15.919859 20541 catalog_manager.cc:4841] TS 5800969806274705b9f893edd0f6a069 (127.6.152.2:43285): tablet 0eb643f059634de3938516f91d5c6fb9 (table test-table [id=64c3f5d6857a442aa7f64bad2f68fdd4]) successfully deleted
I20260928 23:52:15.920042 21053 ts_tablet_manager.cc:1918] T 0eb643f059634de3938516f91d5c6fb9 P ed0f7ba578aa4ba2b5132c9f4741190f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:52:15.920161 21053 log.cc:1196] T 0eb643f059634de3938516f91d5c6fb9 P ed0f7ba578aa4ba2b5132c9f4741190f: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/0eb643f059634de3938516f91d5c6fb9
I20260928 23:52:15.920511 21053 ts_tablet_manager.cc:1939] T 0eb643f059634de3938516f91d5c6fb9 P ed0f7ba578aa4ba2b5132c9f4741190f: Deleting consensus metadata
W20260928 23:52:15.921393 20539 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.152.1:36277 (error 108)
I20260928 23:52:15.921638 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 20750
I20260928 23:52:15.921682 20538 catalog_manager.cc:4841] TS ed0f7ba578aa4ba2b5132c9f4741190f (127.6.152.3:43391): tablet 0eb643f059634de3938516f91d5c6fb9 (table test-table [id=64c3f5d6857a442aa7f64bad2f68fdd4]) successfully deleted
I20260928 23:52:15.922849 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:15.922579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:36277 (local address 127.0.0.1:44902)
0928 23:52:15.922714 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:15.922781 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.1:36277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 23:52:15.923122 20539 catalog_manager.cc:4584] TS 3ddfefce2d894f6697bc7011773750bf (127.6.152.1:36277): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0eb643f059634de3938516f91d5c6fb9: Network error: Client connection negotiation failed: client connection to 127.6.152.1:36277: connect: Connection refused (error 111)
I20260928 23:52:15.927402 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 20884
I20260928 23:52:15.932689 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 20522
I20260928 23:52:15.938611 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:38395
--webserver_interface=127.6.152.62
--webserver_port=33947
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:38395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:52:16.047742 21057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:16.047991 21057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:16.048072 21057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:16.052006 21057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:52:16.052057 21057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:16.052076 21057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:16.052093 21057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:52:16.052148 21057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:52:16.057283 21057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:38395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:38395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=33947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:16.058334 21057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:16.059357 21057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:52:16.072093 21057 server_base.cc:1034] running on GCE node
W20260928 23:52:16.072068 21064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:16.072068 21066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:16.072113 21063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:16.073017 21057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:16.074214 21057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:16.075399 21057 hybrid_clock.cc:648] HybridClock initialized: now 1790639536075389 us; error 30 us; skew 500 ppm
Sep 28 23:52:16 dist-test-slave-p5bw krb5kdc[20509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639536, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:16.078213 21057 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:52:16.079135 21057 webserver.cc:460] Webserver started at http://127.6.152.62:33947/ using document root <none> and password file <none>
I20260928 23:52:16.079759 21057 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:16.079871 21057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:16.083778 21057 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:52:16.085901 21073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:16.087095 21057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:52:16.087246 21057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "d04ec9210a964749a041cf9a3d387ae3"
format_stamp: "Formatted at 2026-09-28 23:52:14 on dist-test-slave-p5bw"
server_key: "1885f4b76284d2205900a43d2567952b"
server_key_iv: "2ea3b37a987ea551b3eb93a9238158dd"
server_key_version: "encryptionkey@0"
I20260928 23:52:16.087839 21057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:16.102828 21057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:16.105652 21057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:16.105859 21057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:16.112576 21057 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:38395
I20260928 23:52:16.112896 21125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:38395 every 8 connection(s)
I20260928 23:52:16.113646 21057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:52:16.114523 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 21057
I20260928 23:52:16.115008 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:36277
--local_ip_for_outbound_sockets=127.6.152.1
--tserver_master_addrs=127.6.152.62:38395
--webserver_port=37833
--webserver_interface=127.6.152.1
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:52:16.118896 21126 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:52:16.120515 21126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3: Bootstrap starting.
I20260928 23:52:16.137362 21126 log.cc:824] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:16.145700 21126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:52:16.146072 21126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3: Bootstrap complete.
I20260928 23:52:16.150197 21126 raft_consensus.cc:348] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04ec9210a964749a041cf9a3d387ae3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38395 } }
I20260928 23:52:16.150573 21126 raft_consensus.cc:729] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d04ec9210a964749a041cf9a3d387ae3, State: Initialized, Role: FOLLOWER
I20260928 23:52:16.151105 21126 consensus_queue.cc:260] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04ec9210a964749a041cf9a3d387ae3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38395 } }
I20260928 23:52:16.151291 21126 raft_consensus.cc:388] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:52:16.151367 21126 raft_consensus.cc:482] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:52:16.151470 21126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:52:16.153568 21126 raft_consensus.cc:504] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04ec9210a964749a041cf9a3d387ae3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38395 } }
I20260928 23:52:16.153913 21126 leader_election.cc:302] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d04ec9210a964749a041cf9a3d387ae3; no voters:
I20260928 23:52:16.154240 21126 leader_election.cc:288] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 23:52:16.154336 21130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 23:52:16.154538 21130 raft_consensus.cc:686] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [term 2 LEADER]: Becoming Leader. State: Replica: d04ec9210a964749a041cf9a3d387ae3, State: Running, Role: LEADER
I20260928 23:52:16.154827 21130 consensus_queue.cc:237] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04ec9210a964749a041cf9a3d387ae3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38395 } }
I20260928 23:52:16.155441 21126 sys_catalog.cc:564] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:52:16.155730 21130 sys_catalog.cc:455] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d04ec9210a964749a041cf9a3d387ae3. Latest consensus state: current_term: 2 leader_uuid: "d04ec9210a964749a041cf9a3d387ae3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04ec9210a964749a041cf9a3d387ae3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38395 } } }
I20260928 23:52:16.155850 21130 sys_catalog.cc:458] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [sys.catalog]: This master's current role is: LEADER
I20260928 23:52:16.156188 21134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:52:16.156225 21131 sys_catalog.cc:455] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d04ec9210a964749a041cf9a3d387ae3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04ec9210a964749a041cf9a3d387ae3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38395 } } }
I20260928 23:52:16.156452 21131 sys_catalog.cc:458] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [sys.catalog]: This master's current role is: LEADER
I20260928 23:52:16.160980 21134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:52:16.161623 21134 catalog_manager.cc:1259] Loaded cluster ID: 3ab7fb17a3d446bdb33ee47962501e5d
I20260928 23:52:16.161695 21134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:52:16.162731 21134 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 23:52:16.165534 21134 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 23:52:16.169991 21134 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:52:16.170709 21134 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3: Loaded TSK: 0
I20260928 23:52:16.171978 21134 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:52:16.244388 21128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:16.244658 21128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:16.244702 21128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:16.248764 21128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:16.248857 21128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:16.248961 21128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:52:16.254317 21128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:36277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=37833
--tserver_master_addrs=127.6.152.62:38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:16.255381 21128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:16.256559 21128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:16.267104 21153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:16.267174 21128 server_base.cc:1034] running on GCE node
W20260928 23:52:16.267120 21154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:16.267362 21156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:16.267709 21128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:16.268463 21128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:16.269670 21128 hybrid_clock.cc:648] HybridClock initialized: now 1790639536269634 us; error 52 us; skew 500 ppm
Sep 28 23:52:16 dist-test-slave-p5bw krb5kdc[20509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639536, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:16.272482 21128 init.cc:377] Logged in from keytab as kudu/127.6.152.1@KRBTEST.COM (short username kudu)
I20260928 23:52:16.273478 21128 webserver.cc:460] Webserver started at http://127.6.152.1:37833/ using document root <none> and password file <none>
I20260928 23:52:16.274035 21128 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:16.274107 21128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:16.278046 21128 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:52:16.280192 21163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:16.281566 21128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:52:16.281697 21128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "3ddfefce2d894f6697bc7011773750bf"
format_stamp: "Formatted at 2026-09-28 23:52:15 on dist-test-slave-p5bw"
server_key: "3aedbeb410904fbcba587d29f530446f"
server_key_iv: "28fe90867f765bd5a57dfdcafa757337"
server_key_version: "encryptionkey@0"
I20260928 23:52:16.282226 21128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:16.297062 21128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:16.300292 21128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:16.300503 21128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:16.301183 21128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:52:16.302385 21170 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 23:52:16.307832 21128 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 23:52:16.307940 21128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260928 23:52:16.308025 21128 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 23:52:16.309842 21128 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 23:52:16.309921 21128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260928 23:52:16.310258 21170 tablet_bootstrap.cc:492] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf: Bootstrap starting.
I20260928 23:52:16.320533 21128 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:36277
I20260928 23:52:16.320837 21277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:36277 every 8 connection(s)
I20260928 23:52:16.321715 21128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:52:16.331883 21170 log.cc:824] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:16.331949 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 21128
Sep 28 23:52:16 dist-test-slave-p5bw krb5kdc[20509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639536, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:52:16.332445 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:43285
--local_ip_for_outbound_sockets=127.6.152.2
--tserver_master_addrs=127.6.152.62:38395
--webserver_port=35505
--webserver_interface=127.6.152.2
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:52:16.338629 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:16.323682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:39807 (local address 127.6.152.62:38395)
0928 23:52:16.324769 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:16.324799 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:16.324890 (+ 91us) server_negotiation.cc:407] Connection header received
0928 23:52:16.325628 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:16.325638 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:16.325927 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:16.326247 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:16.327336 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.328293 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.329542 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.330010 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.330193 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:16.333293 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:16.333302 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:16.333323 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:16.333363 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:16.335578 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:16.336200 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:16.336204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:16.336208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:16.336340 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:16.336685 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:16.336687 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:16.336688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:16.337065 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:16.337167 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:16.337756 (+ 589us) server_negotiation.cc:299] Negotiation successful
0928 23:52:16.337984 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":950,"thread_start_us":102,"threads_started":1}
I20260928 23:52:16.338795 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:16.323729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.1:39807)
0928 23:52:16.324223 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:16.324276 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:16.325222 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:16.326313 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:16.326321 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:16.326725 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:16.327165 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:16.327178 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.328453 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.328456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:16.329413 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:16.329423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.330154 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.330160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:16.330356 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:16.330952 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:16.330969 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:16.333064 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:16.335732 (+ 2668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:16.335737 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:16.335750 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:16.336074 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:16.336431 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:16.336434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:16.336436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:16.336594 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:16.337171 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:16.337177 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:16.337468 (+ 291us) client_negotiation.cc:769] Sending connection context
0928 23:52:16.337654 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 23:52:16.337978 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":121,"threads_started":1}
I20260928 23:52:16.339835 21170 tablet_bootstrap.cc:492] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:52:16.340193 21170 tablet_bootstrap.cc:492] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf: Bootstrap complete.
I20260928 23:52:16.341614 21170 ts_tablet_manager.cc:1397] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf: Time spent bootstrapping tablet: real 0.032s user 0.018s sys 0.012s
I20260928 23:52:16.341660 21278 heartbeater.cc:347] Connected to a master server at 127.6.152.62:38395
I20260928 23:52:16.341922 21278 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:16.342535 21278 heartbeater.cc:511] Master 127.6.152.62:38395 requested a full tablet report, sending...
I20260928 23:52:16.344085 21170 raft_consensus.cc:348] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } }
I20260928 23:52:16.344587 21170 raft_consensus.cc:729] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ddfefce2d894f6697bc7011773750bf, State: Initialized, Role: FOLLOWER
I20260928 23:52:16.345085 21170 consensus_queue.cc:260] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } }
I20260928 23:52:16.345430 21090 ts_manager.cc:194] Registered new tserver with Master: 3ddfefce2d894f6697bc7011773750bf (127.6.152.1:36277)
I20260928 23:52:16.346098 21170 ts_tablet_manager.cc:1428] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf: Time spent starting tablet: real 0.004s user 0.003s sys 0.002s
I20260928 23:52:16.348835 21090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.1@KRBTEST.COM'} at 127.6.152.1:39807
I20260928 23:52:16.351337 21278 heartbeater.cc:503] Master 127.6.152.62:38395 was elected leader, sending a full tablet report...
Sep 28 23:52:16 dist-test-slave-p5bw krb5kdc[20509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639536, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for kudu/127.6.152.1@KRBTEST.COM
I20260928 23:52:16.358383 21285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:16.348318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:36277 (local address 127.0.0.1:44914)
0928 23:52:16.348984 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:16.349025 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:16.349159 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:16.349688 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:16.349694 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:16.350044 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:16.350139 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:16.350146 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.351070 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.351076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:16.351827 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:16.351837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.352384 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.352388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:16.352499 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:16.353102 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:16.353113 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:16.354578 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:16.356526 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:16.356534 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:16.356539 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:16.356857 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:16.357133 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:16.357138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:16.357140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:16.357233 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:16.357722 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:16.357729 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:16.357872 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 23:52:16.358004 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 23:52:16.358195 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":117,"threads_started":1}
I20260928 23:52:16.358392 21286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:16.348376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44914 (local address 127.6.152.1:36277)
0928 23:52:16.348716 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:16.348727 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:16.349125 (+ 398us) server_negotiation.cc:407] Connection header received
0928 23:52:16.349305 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:16.349310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:16.349499 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:16.349625 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:16.350262 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.350956 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.351936 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.352268 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.352384 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:16.354729 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:16.354738 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:16.354747 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:16.354767 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:16.356388 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:16.356951 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:16.356956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:16.356958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:16.357040 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:16.357327 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:16.357330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:16.357331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:16.357604 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:16.357754 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:16.358096 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 23:52:16.358253 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":105,"threads_started":1}
I20260928 23:52:16.359362 21212 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eb643f059634de3938516f91d5c6fb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:52:15 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:44914
I20260928 23:52:16.360316 21287 tablet_replica.cc:330] stopping tablet replica
I20260928 23:52:16.360598 21287 raft_consensus.cc:2229] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:52:16.360904 21287 raft_consensus.cc:2258] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:52:16.361831 21287 ts_tablet_manager.cc:1905] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:52:16.365350 21287 ts_tablet_manager.cc:1918] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:52:16.365461 21287 log.cc:1196] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/wal/wals/0eb643f059634de3938516f91d5c6fb9
I20260928 23:52:16.365734 21287 ts_tablet_manager.cc:1939] T 0eb643f059634de3938516f91d5c6fb9 P 3ddfefce2d894f6697bc7011773750bf: Deleting consensus metadata
I20260928 23:52:16.367317 21075 catalog_manager.cc:4841] TS 3ddfefce2d894f6697bc7011773750bf (127.6.152.1:36277): tablet 0eb643f059634de3938516f91d5c6fb9 (table test-table [id=64c3f5d6857a442aa7f64bad2f68fdd4]) successfully deleted
W20260928 23:52:16.453253 21283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:16.453495 21283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:16.453536 21283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:16.457697 21283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:16.457749 21283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:16.457839 21283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:52:16.463178 21283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:43285
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=35505
--tserver_master_addrs=127.6.152.62:38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:16.464156 21283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:16.465157 21283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:16.474534 21293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:16.474555 21292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:16.474555 21295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:16.474856 21283 server_base.cc:1034] running on GCE node
I20260928 23:52:16.475288 21283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:16.476019 21283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:16.477198 21283 hybrid_clock.cc:648] HybridClock initialized: now 1790639536477186 us; error 25 us; skew 500 ppm
Sep 28 23:52:16 dist-test-slave-p5bw krb5kdc[20509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639536, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:16.479851 21283 init.cc:377] Logged in from keytab as kudu/127.6.152.2@KRBTEST.COM (short username kudu)
I20260928 23:52:16.480665 21283 webserver.cc:460] Webserver started at http://127.6.152.2:35505/ using document root <none> and password file <none>
I20260928 23:52:16.481312 21283 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:16.481388 21283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:16.485070 21283 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 23:52:16.487088 21302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:16.488274 21283 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:52:16.488363 21283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "5800969806274705b9f893edd0f6a069"
format_stamp: "Formatted at 2026-09-28 23:52:15 on dist-test-slave-p5bw"
server_key: "1f45295d76092946fe0cd5ea6b7a8a9d"
server_key_iv: "b444aa4e08a90c67bcf9b8b1ac81b01a"
server_key_version: "encryptionkey@0"
I20260928 23:52:16.488907 21283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:16.503250 21283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:16.506250 21283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:16.506392 21283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:16.506871 21283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:52:16.507592 21283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:52:16.507637 21283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:16.507675 21283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:52:16.507691 21283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:16.516484 21283 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:43285
I20260928 23:52:16.516511 21415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:43285 every 8 connection(s)
I20260928 23:52:16.517441 21283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:52:16.524328 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 21283
I20260928 23:52:16.524883 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:43391
--local_ip_for_outbound_sockets=127.6.152.3
--tserver_master_addrs=127.6.152.62:38395
--webserver_port=35319
--webserver_interface=127.6.152.3
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:52:16 dist-test-slave-p5bw krb5kdc[20509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639536, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:52:16.532037 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:16.519243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:43799 (local address 127.6.152.62:38395)
0928 23:52:16.519375 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:16.519385 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:16.520141 (+ 756us) server_negotiation.cc:407] Connection header received
0928 23:52:16.521538 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:16.521542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:16.521605 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:16.521705 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:16.522828 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.523369 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.524749 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.525110 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.525208 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:16.527835 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:16.527841 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:16.527847 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:16.527883 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:16.529785 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:16.530284 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:16.530289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:16.530294 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:16.530397 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:16.530734 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:16.530739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:16.530743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:16.531023 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:16.531137 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:16.531617 (+ 480us) server_negotiation.cc:299] Negotiation successful
0928 23:52:16.531797 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:52:16.532655 21418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:16.519572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.2:43799)
0928 23:52:16.519997 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:16.520037 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:16.521335 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:16.521885 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:16.521891 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:16.522228 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:16.522682 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:16.522694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.523492 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.523496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:16.524589 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:16.524602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.525266 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.525272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:16.525528 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:16.526125 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:16.526150 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:16.527652 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:16.529906 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:16.529911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:16.529923 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:16.530174 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:16.530498 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:16.530500 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:16.530502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:16.530630 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:16.531135 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:16.531140 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:16.531379 (+ 239us) client_negotiation.cc:769] Sending connection context
0928 23:52:16.531531 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 23:52:16.531793 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":123,"threads_started":1}
I20260928 23:52:16.533864 21416 heartbeater.cc:347] Connected to a master server at 127.6.152.62:38395
I20260928 23:52:16.534143 21416 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:16.534801 21416 heartbeater.cc:511] Master 127.6.152.62:38395 requested a full tablet report, sending...
I20260928 23:52:16.535986 21090 ts_manager.cc:194] Registered new tserver with Master: 5800969806274705b9f893edd0f6a069 (127.6.152.2:43285)
I20260928 23:52:16.536554 21090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.2@KRBTEST.COM'} at 127.6.152.2:43799
W20260928 23:52:16.638247 21419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:16.638515 21419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:16.638559 21419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:16.643184 21419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:16.643255 21419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:16.643356 21419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:52:16.652380 21419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:43391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=35319
--tserver_master_addrs=127.6.152.62:38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:16.653605 21419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:16.654625 21419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:16.665469 21424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:16.665517 21427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:16.665474 21425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:16.665663 21419 server_base.cc:1034] running on GCE node
I20260928 23:52:16.666168 21419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:16.666785 21419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:16.667968 21419 hybrid_clock.cc:648] HybridClock initialized: now 1790639536667950 us; error 29 us; skew 500 ppm
Sep 28 23:52:16 dist-test-slave-p5bw krb5kdc[20509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639536, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:16.670715 21419 init.cc:377] Logged in from keytab as kudu/127.6.152.3@KRBTEST.COM (short username kudu)
I20260928 23:52:16.671595 21419 webserver.cc:460] Webserver started at http://127.6.152.3:35319/ using document root <none> and password file <none>
I20260928 23:52:16.672181 21419 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:16.672272 21419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:16.676272 21419 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:52:16.678181 21434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:16.679373 21419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:52:16.679492 21419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "ed0f7ba578aa4ba2b5132c9f4741190f"
format_stamp: "Formatted at 2026-09-28 23:52:15 on dist-test-slave-p5bw"
server_key: "4e0d9997cbb1330a432433630f406c41"
server_key_iv: "6eb8620012a4a37648682c2e55361d3f"
server_key_version: "encryptionkey@0"
I20260928 23:52:16.680024 21419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:16.730232 21419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:16.733525 21419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:16.733726 21419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:16.734319 21419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:52:16.735102 21419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:52:16.735146 21419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:16.735213 21419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:52:16.735251 21419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:16.743811 21419 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:43391
I20260928 23:52:16.743844 21547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:43391 every 8 connection(s)
I20260928 23:52:16.744764 21419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:52:16.752274 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 21419
Sep 28 23:52:16 dist-test-slave-p5bw krb5kdc[20509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639536, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:52:16.759030 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:16.746528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:55807 (local address 127.6.152.62:38395)
0928 23:52:16.746675 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:16.746678 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:16.747451 (+ 773us) server_negotiation.cc:407] Connection header received
0928 23:52:16.748745 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:16.748749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:16.748804 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:16.748948 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:16.750130 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.750655 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.751797 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.752026 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.752139 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:16.755155 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:16.755162 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:16.755165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:16.755196 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:16.756802 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:16.757410 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:16.757417 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:16.757422 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:16.757520 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:16.757929 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:16.757932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:16.757933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:16.758092 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:16.758170 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:16.758747 (+ 577us) server_negotiation.cc:299] Negotiation successful
0928 23:52:16.758861 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:52:16.759920 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:16.746882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.3:55807)
0928 23:52:16.747300 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:16.747341 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:16.748524 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:16.749127 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:16.749135 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:16.749499 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:16.749978 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:16.749991 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.750786 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.750791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:16.751662 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:16.751671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.752177 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.752181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:16.752392 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:16.753266 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:16.753287 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:16.754916 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:16.756916 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:16.756922 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:16.756937 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:16.757272 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:16.757628 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:16.757634 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:16.757638 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:16.757825 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:16.758193 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:16.758199 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:16.758498 (+ 299us) client_negotiation.cc:769] Sending connection context
0928 23:52:16.758677 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 23:52:16.758995 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":113,"threads_started":1}
I20260928 23:52:16.761078 21548 heartbeater.cc:347] Connected to a master server at 127.6.152.62:38395
I20260928 23:52:16.761327 21548 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:16.761706 21552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:16.752994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44622 (local address 127.6.152.62:38395)
0928 23:52:16.753271 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:16.753276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:16.753414 (+ 138us) server_negotiation.cc:407] Connection header received
0928 23:52:16.753547 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:16.753549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:16.753585 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:16.753658 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:16.754497 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.755118 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.756303 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.756709 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.756853 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:16.758411 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:16.758414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:16.758415 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:16.758432 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:16.759955 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:16.760466 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:16.760473 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:16.760476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:16.760565 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:16.760870 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:16.760874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:16.760876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:16.761047 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:16.761264 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:16.761435 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 23:52:16.761550 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":98,"threads_started":1}
I20260928 23:52:16.761919 21548 heartbeater.cc:511] Master 127.6.152.62:38395 requested a full tablet report, sending...
I20260928 23:52:16.763029 21089 ts_manager.cc:194] Registered new tserver with Master: ed0f7ba578aa4ba2b5132c9f4741190f (127.6.152.3:43391)
I20260928 23:52:16.763659 21089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.3@KRBTEST.COM'} at 127.6.152.3:55807
I20260928 23:52:16.775105 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:52:16.779727 21552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:16.775767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44636 (local address 127.6.152.62:38395)
0928 23:52:16.775896 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:16.775900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:16.776207 (+ 307us) server_negotiation.cc:407] Connection header received
0928 23:52:16.776361 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:16.776363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:16.776462 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:16.776548 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:52:16.776806 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.777377 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.778343 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.778565 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.778641 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:16.778950 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:52:16.779350 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:52:16.779437 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:16.779568 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 23:52:16.779618 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:52:16.780413 21089 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:52:16.783957 21089 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:52:16.791946 21212 tablet_service.cc:1467] Processing CreateTablet for tablet c96714228ad043e18499097af9fe1257 (DEFAULT_TABLE table=test-table [id=9504c16bf3b345e1997e00ed2920bda9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:52:16.792644 21212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c96714228ad043e18499097af9fe1257. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:52:16.795341 21170 tablet_bootstrap.cc:492] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf: Bootstrap starting.
I20260928 23:52:16.796207 21557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:16.790947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:43285 (local address 127.0.0.1:41684)
0928 23:52:16.791199 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:16.791214 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:16.791329 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:16.792249 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:16.792254 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:16.792274 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:52:16.792395 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:16.792401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.793708 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.793711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:16.794486 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:16.794494 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.795769 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.795773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:16.795863 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:16.795921 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 23:52:16.795991 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 23:52:16.796072 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":119,"threads_started":1}
I20260928 23:52:16.796226 21170 tablet_bootstrap.cc:654] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf: Neither blocks nor log segments found. Creating new log.
I20260928 23:52:16.796638 21558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:16.791391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41684 (local address 127.6.152.2:43285)
0928 23:52:16.791757 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:16.791769 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:16.791795 (+ 26us) server_negotiation.cc:407] Connection header received
0928 23:52:16.791871 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:16.791877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:16.792093 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:16.792260 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:52:16.792494 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.793572 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.794613 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.795438 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.795542 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:16.795697 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:16.796372 (+ 675us) server_negotiation.cc:299] Negotiation successful
0928 23:52:16.796475 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":105,"threads_started":1}
I20260928 23:52:16.797802 21170 tablet_bootstrap.cc:492] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf: No bootstrap required, opened a new log
I20260928 23:52:16.797888 21170 ts_tablet_manager.cc:1397] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 23:52:16.798336 21170 raft_consensus.cc:348] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } }
I20260928 23:52:16.798465 21170 raft_consensus.cc:374] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:52:16.798509 21170 raft_consensus.cc:729] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ddfefce2d894f6697bc7011773750bf, State: Initialized, Role: FOLLOWER
I20260928 23:52:16.798656 21170 consensus_queue.cc:260] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } }
I20260928 23:52:16.799342 21170 ts_tablet_manager.cc:1428] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 23:52:16.799847 21350 tablet_service.cc:1467] Processing CreateTablet for tablet c96714228ad043e18499097af9fe1257 (DEFAULT_TABLE table=test-table [id=9504c16bf3b345e1997e00ed2920bda9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:52:16.800987 21350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c96714228ad043e18499097af9fe1257. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:52:16.801236 21285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:16.790691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:43391 (local address 127.0.0.1:34378)
0928 23:52:16.790827 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:16.790841 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:16.790921 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:16.796615 (+ 5694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:16.796620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:16.796644 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:52:16.796739 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:16.796745 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.798274 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.798278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:16.799382 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:16.799393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.800798 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.800801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:16.800923 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:16.800946 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:52:16.801029 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 23:52:16.801110 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 23:52:16.801294 21559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:16.795503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34378 (local address 127.6.152.3:43391)
0928 23:52:16.795868 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:16.795887 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:16.795912 (+ 25us) server_negotiation.cc:407] Connection header received
0928 23:52:16.795983 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:16.795988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:16.796197 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:16.796996 (+ 799us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:52:16.797159 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.798132 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.799644 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.800669 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.800770 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:16.800930 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:16.801060 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 23:52:16.801154 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":147,"threads_started":1}
I20260928 23:52:16.804129 21482 tablet_service.cc:1467] Processing CreateTablet for tablet c96714228ad043e18499097af9fe1257 (DEFAULT_TABLE table=test-table [id=9504c16bf3b345e1997e00ed2920bda9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:52:16.805001 21482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c96714228ad043e18499097af9fe1257. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:52:16.807981 21561 tablet_bootstrap.cc:492] T c96714228ad043e18499097af9fe1257 P 5800969806274705b9f893edd0f6a069: Bootstrap starting.
I20260928 23:52:16.810624 21561 tablet_bootstrap.cc:654] T c96714228ad043e18499097af9fe1257 P 5800969806274705b9f893edd0f6a069: Neither blocks nor log segments found. Creating new log.
I20260928 23:52:16.811474 21561 log.cc:824] T c96714228ad043e18499097af9fe1257 P 5800969806274705b9f893edd0f6a069: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:16.812474 21562 tablet_bootstrap.cc:492] T c96714228ad043e18499097af9fe1257 P ed0f7ba578aa4ba2b5132c9f4741190f: Bootstrap starting.
I20260928 23:52:16.813511 21561 tablet_bootstrap.cc:492] T c96714228ad043e18499097af9fe1257 P 5800969806274705b9f893edd0f6a069: No bootstrap required, opened a new log
I20260928 23:52:16.813696 21561 ts_tablet_manager.cc:1397] T c96714228ad043e18499097af9fe1257 P 5800969806274705b9f893edd0f6a069: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260928 23:52:16.814549 21562 tablet_bootstrap.cc:654] T c96714228ad043e18499097af9fe1257 P ed0f7ba578aa4ba2b5132c9f4741190f: Neither blocks nor log segments found. Creating new log.
I20260928 23:52:16.815706 21562 log.cc:824] T c96714228ad043e18499097af9fe1257 P ed0f7ba578aa4ba2b5132c9f4741190f: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:16.817553 21562 tablet_bootstrap.cc:492] T c96714228ad043e18499097af9fe1257 P ed0f7ba578aa4ba2b5132c9f4741190f: No bootstrap required, opened a new log
I20260928 23:52:16.817726 21562 ts_tablet_manager.cc:1397] T c96714228ad043e18499097af9fe1257 P ed0f7ba578aa4ba2b5132c9f4741190f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:52:16.817525 21561 raft_consensus.cc:348] T c96714228ad043e18499097af9fe1257 P 5800969806274705b9f893edd0f6a069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } }
I20260928 23:52:16.817798 21561 raft_consensus.cc:374] T c96714228ad043e18499097af9fe1257 P 5800969806274705b9f893edd0f6a069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:52:16.817862 21561 raft_consensus.cc:729] T c96714228ad043e18499097af9fe1257 P 5800969806274705b9f893edd0f6a069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5800969806274705b9f893edd0f6a069, State: Initialized, Role: FOLLOWER
I20260928 23:52:16.818301 21561 consensus_queue.cc:260] T c96714228ad043e18499097af9fe1257 P 5800969806274705b9f893edd0f6a069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } }
I20260928 23:52:16.818910 21416 heartbeater.cc:503] Master 127.6.152.62:38395 was elected leader, sending a full tablet report...
I20260928 23:52:16.819284 21561 ts_tablet_manager.cc:1428] T c96714228ad043e18499097af9fe1257 P 5800969806274705b9f893edd0f6a069: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:52:16.820779 21562 raft_consensus.cc:348] T c96714228ad043e18499097af9fe1257 P ed0f7ba578aa4ba2b5132c9f4741190f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } }
I20260928 23:52:16.821038 21562 raft_consensus.cc:374] T c96714228ad043e18499097af9fe1257 P ed0f7ba578aa4ba2b5132c9f4741190f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:52:16.821120 21562 raft_consensus.cc:729] T c96714228ad043e18499097af9fe1257 P ed0f7ba578aa4ba2b5132c9f4741190f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed0f7ba578aa4ba2b5132c9f4741190f, State: Initialized, Role: FOLLOWER
I20260928 23:52:16.821520 21562 consensus_queue.cc:260] T c96714228ad043e18499097af9fe1257 P ed0f7ba578aa4ba2b5132c9f4741190f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } }
I20260928 23:52:16.822109 21548 heartbeater.cc:503] Master 127.6.152.62:38395 was elected leader, sending a full tablet report...
I20260928 23:52:16.822350 21562 ts_tablet_manager.cc:1428] T c96714228ad043e18499097af9fe1257 P ed0f7ba578aa4ba2b5132c9f4741190f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260928 23:52:16.823024 21279 tablet.cc:2369] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:52:16.920262 21284 raft_consensus.cc:482] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:52:16.920537 21284 raft_consensus.cc:504] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } }
I20260928 23:52:16.921676 21284 leader_election.cc:288] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed0f7ba578aa4ba2b5132c9f4741190f (127.6.152.3:43391), 5800969806274705b9f893edd0f6a069 (127.6.152.2:43285)
I20260928 23:52:16.925607 21559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:16.922052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:40281 (local address 127.6.152.3:43391)
0928 23:52:16.922178 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:16.922183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:16.922429 (+ 246us) server_negotiation.cc:407] Connection header received
0928 23:52:16.922546 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:16.922549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:16.922590 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:16.922677 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:52:16.922981 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.923463 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.924559 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.924985 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.925064 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:16.925137 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:16.925411 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 23:52:16.925480 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:52:16.925606 21558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:16.922103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:55769 (local address 127.6.152.2:43285)
0928 23:52:16.922248 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:16.922254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:16.922302 (+ 48us) server_negotiation.cc:407] Connection header received
0928 23:52:16.922516 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:16.922522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:16.922579 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:16.922637 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:52:16.922947 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.923465 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.924598 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.925067 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.925165 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:16.925223 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:16.925410 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 23:52:16.925476 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 23:52:16.925628 21567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:16.921960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:43391 (local address 127.6.152.1:40281)
0928 23:52:16.922316 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:16.922332 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:16.922426 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:16.922708 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:16.922711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:16.922741 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:52:16.922849 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:16.922854 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.923573 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.923579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:16.924435 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:16.924444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.925099 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.925106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:16.925212 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:16.925305 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 23:52:16.925393 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 23:52:16.925483 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":69,"thread_start_us":79,"threads_started":1}
I20260928 23:52:16.925647 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:16.921911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:43285 (local address 127.6.152.1:55769)
0928 23:52:16.922206 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:16.922221 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:16.922349 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:16.922676 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:16.922679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:16.922694 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:52:16.922839 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:16.922845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.923597 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.923600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:16.924445 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:16.924454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.925189 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.925194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:16.925284 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:16.925305 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 23:52:16.925395 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 23:52:16.925497 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":89,"spinlock_wait_cycles":19200,"thread_start_us":73,"threads_started":1}
I20260928 23:52:16.926219 21502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c96714228ad043e18499097af9fe1257" candidate_uuid: "3ddfefce2d894f6697bc7011773750bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" is_pre_election: true
I20260928 23:52:16.926229 21370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c96714228ad043e18499097af9fe1257" candidate_uuid: "3ddfefce2d894f6697bc7011773750bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5800969806274705b9f893edd0f6a069" is_pre_election: true
I20260928 23:52:16.926498 21502 raft_consensus.cc:2454] T c96714228ad043e18499097af9fe1257 P ed0f7ba578aa4ba2b5132c9f4741190f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ddfefce2d894f6697bc7011773750bf in term 0.
I20260928 23:52:16.926524 21370 raft_consensus.cc:2454] T c96714228ad043e18499097af9fe1257 P 5800969806274705b9f893edd0f6a069 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ddfefce2d894f6697bc7011773750bf in term 0.
I20260928 23:52:16.926928 21164 leader_election.cc:302] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ddfefce2d894f6697bc7011773750bf, ed0f7ba578aa4ba2b5132c9f4741190f; no voters:
I20260928 23:52:16.927174 21284 raft_consensus.cc:2781] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:52:16.927256 21284 raft_consensus.cc:482] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:52:16.927306 21284 raft_consensus.cc:3037] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:52:16.928315 21284 raft_consensus.cc:504] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } }
I20260928 23:52:16.928651 21284 leader_election.cc:288] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 1 election: Requested vote from peers ed0f7ba578aa4ba2b5132c9f4741190f (127.6.152.3:43391), 5800969806274705b9f893edd0f6a069 (127.6.152.2:43285)
I20260928 23:52:16.928972 21502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c96714228ad043e18499097af9fe1257" candidate_uuid: "3ddfefce2d894f6697bc7011773750bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f"
I20260928 23:52:16.929092 21502 raft_consensus.cc:3037] T c96714228ad043e18499097af9fe1257 P ed0f7ba578aa4ba2b5132c9f4741190f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:52:16.929060 21370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c96714228ad043e18499097af9fe1257" candidate_uuid: "3ddfefce2d894f6697bc7011773750bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5800969806274705b9f893edd0f6a069"
I20260928 23:52:16.929227 21370 raft_consensus.cc:3037] T c96714228ad043e18499097af9fe1257 P 5800969806274705b9f893edd0f6a069 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:52:16.930151 21502 raft_consensus.cc:2454] T c96714228ad043e18499097af9fe1257 P ed0f7ba578aa4ba2b5132c9f4741190f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ddfefce2d894f6697bc7011773750bf in term 1.
I20260928 23:52:16.930223 21370 raft_consensus.cc:2454] T c96714228ad043e18499097af9fe1257 P 5800969806274705b9f893edd0f6a069 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ddfefce2d894f6697bc7011773750bf in term 1.
I20260928 23:52:16.930415 21164 leader_election.cc:302] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ddfefce2d894f6697bc7011773750bf, ed0f7ba578aa4ba2b5132c9f4741190f; no voters:
I20260928 23:52:16.930613 21284 raft_consensus.cc:2781] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:52:16.930830 21284 raft_consensus.cc:686] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 1 LEADER]: Becoming Leader. State: Replica: 3ddfefce2d894f6697bc7011773750bf, State: Running, Role: LEADER
I20260928 23:52:16.931082 21284 consensus_queue.cc:237] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } }
I20260928 23:52:16.933959 21089 catalog_manager.cc:5495] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ddfefce2d894f6697bc7011773750bf (127.6.152.1). New cstate: current_term: 1 leader_uuid: "3ddfefce2d894f6697bc7011773750bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } health_report { overall_health: HEALTHY } } }
I20260928 23:52:16.952792 21571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:16.949044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44930 (local address 127.6.152.1:36277)
0928 23:52:16.949381 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:16.949389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:16.949400 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:52:16.949469 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:16.949472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:16.949523 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:16.949667 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:52:16.949900 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.950446 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.951282 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:16.951522 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:16.951640 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:16.951922 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:52:16.952369 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:52:16.952502 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:16.952562 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 23:52:16.952618 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":119,"threads_started":1}
I20260928 23:52:16.957020 21502 raft_consensus.cc:1261] T c96714228ad043e18499097af9fe1257 P ed0f7ba578aa4ba2b5132c9f4741190f [term 1 FOLLOWER]: Refusing update from remote peer 3ddfefce2d894f6697bc7011773750bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:52:16.957067 21370 raft_consensus.cc:1261] T c96714228ad043e18499097af9fe1257 P 5800969806274705b9f893edd0f6a069 [term 1 FOLLOWER]: Refusing update from remote peer 3ddfefce2d894f6697bc7011773750bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:52:16.957795 21284 consensus_queue.cc:1035] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:52:16.958127 21569 consensus_queue.cc:1035] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:52:16.967949 21573 mvcc.cc:205] Tried to move back new op lower bound from 7334459543368302592 to 7334459543272366080. Current Snapshot: MvccSnapshot[applied={T|T < 7334459543368302592}]
I20260928 23:52:16.975556 21090 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:52:16.975751 21090 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:52:16.978196 21090 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3: Sending DeleteTablet for 3 replicas of tablet c96714228ad043e18499097af9fe1257
I20260928 23:52:16.978910 21482 tablet_service.cc:1514] Processing DeleteTablet for tablet c96714228ad043e18499097af9fe1257 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:52:16 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:34378
I20260928 23:52:16.978932 21350 tablet_service.cc:1514] Processing DeleteTablet for tablet c96714228ad043e18499097af9fe1257 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:52:16 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:41684
I20260928 23:52:16.979427 21581 tablet_replica.cc:330] T c96714228ad043e18499097af9fe1257 P ed0f7ba578aa4ba2b5132c9f4741190f: stopping tablet replica
I20260928 23:52:16.979532 21212 tablet_service.cc:1514] Processing DeleteTablet for tablet c96714228ad043e18499097af9fe1257 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:52:16 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:44914
I20260928 23:52:16.979753 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 21128
I20260928 23:52:16.979885 21583 tablet_replica.cc:330] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf: stopping tablet replica
I20260928 23:52:16.979920 21581 raft_consensus.cc:2229] T c96714228ad043e18499097af9fe1257 P ed0f7ba578aa4ba2b5132c9f4741190f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:52:16.980011 21583 raft_consensus.cc:2229] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:52:16.980376 21581 raft_consensus.cc:2258] T c96714228ad043e18499097af9fe1257 P ed0f7ba578aa4ba2b5132c9f4741190f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:52:16.980535 21582 tablet_replica.cc:330] T c96714228ad043e18499097af9fe1257 P 5800969806274705b9f893edd0f6a069: stopping tablet replica
I20260928 23:52:16.980885 21582 raft_consensus.cc:2229] T c96714228ad043e18499097af9fe1257 P 5800969806274705b9f893edd0f6a069 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:52:16.981187 21582 raft_consensus.cc:2258] T c96714228ad043e18499097af9fe1257 P 5800969806274705b9f893edd0f6a069 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:52:16.982146 21581 ts_tablet_manager.cc:1905] T c96714228ad043e18499097af9fe1257 P ed0f7ba578aa4ba2b5132c9f4741190f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:52:16.982730 21582 ts_tablet_manager.cc:1905] T c96714228ad043e18499097af9fe1257 P 5800969806274705b9f893edd0f6a069: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:52:16.985186 21582 ts_tablet_manager.cc:1918] T c96714228ad043e18499097af9fe1257 P 5800969806274705b9f893edd0f6a069: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:52:16.985318 21582 log.cc:1196] T c96714228ad043e18499097af9fe1257 P 5800969806274705b9f893edd0f6a069: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/wal/wals/c96714228ad043e18499097af9fe1257
I20260928 23:52:16.985643 21582 ts_tablet_manager.cc:1939] T c96714228ad043e18499097af9fe1257 P 5800969806274705b9f893edd0f6a069: Deleting consensus metadata
I20260928 23:52:16.985682 21581 ts_tablet_manager.cc:1918] T c96714228ad043e18499097af9fe1257 P ed0f7ba578aa4ba2b5132c9f4741190f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:52:16.985791 21581 log.cc:1196] T c96714228ad043e18499097af9fe1257 P ed0f7ba578aa4ba2b5132c9f4741190f: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/c96714228ad043e18499097af9fe1257
I20260928 23:52:16.986123 21581 ts_tablet_manager.cc:1939] T c96714228ad043e18499097af9fe1257 P ed0f7ba578aa4ba2b5132c9f4741190f: Deleting consensus metadata
I20260928 23:52:16.986963 21077 catalog_manager.cc:4841] TS 5800969806274705b9f893edd0f6a069 (127.6.152.2:43285): tablet c96714228ad043e18499097af9fe1257 (table test-table [id=9504c16bf3b345e1997e00ed2920bda9]) successfully deleted
I20260928 23:52:16.987227 21074 catalog_manager.cc:4841] TS ed0f7ba578aa4ba2b5132c9f4741190f (127.6.152.3:43391): tablet c96714228ad043e18499097af9fe1257 (table test-table [id=9504c16bf3b345e1997e00ed2920bda9]) successfully deleted
W20260928 23:52:16.988260 21075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.152.1:36277 (error 108)
I20260928 23:52:16.989074 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 21283
I20260928 23:52:16.989320 21285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:16.989002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:36277 (local address 127.0.0.1:44940)
0928 23:52:16.989147 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:16.989225 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.1:36277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 23:52:16.992939 21075 catalog_manager.cc:4584] TS 3ddfefce2d894f6697bc7011773750bf (127.6.152.1:36277): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c96714228ad043e18499097af9fe1257: Network error: Client connection negotiation failed: client connection to 127.6.152.1:36277: connect: Connection refused (error 111)
I20260928 23:52:16.994968 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 21419
I20260928 23:52:17.000653 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 21057
I20260928 23:52:17.007306 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:38395
--webserver_interface=127.6.152.62
--webserver_port=33947
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:38395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:52:17.114908 21585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:17.115236 21585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:17.115324 21585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:17.119329 21585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:52:17.119426 21585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:17.119463 21585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:17.119521 21585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:52:17.119557 21585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:52:17.124933 21585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:38395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:38395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=33947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:17.125953 21585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:17.126947 21585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:17.135283 21591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:17.135310 21590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:17.135317 21593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:17.135370 21585 server_base.cc:1034] running on GCE node
I20260928 23:52:17.136135 21585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:17.137214 21585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:17.138376 21585 hybrid_clock.cc:648] HybridClock initialized: now 1790639537138341 us; error 48 us; skew 500 ppm
Sep 28 23:52:17 dist-test-slave-p5bw krb5kdc[20509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639537, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:17.141029 21585 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:52:17.141842 21585 webserver.cc:460] Webserver started at http://127.6.152.62:33947/ using document root <none> and password file <none>
I20260928 23:52:17.142372 21585 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:17.142431 21585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:17.146077 21585 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20260928 23:52:17.148075 21600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:17.149302 21585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:52:17.149430 21585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "d04ec9210a964749a041cf9a3d387ae3"
format_stamp: "Formatted at 2026-09-28 23:52:14 on dist-test-slave-p5bw"
server_key: "1885f4b76284d2205900a43d2567952b"
server_key_iv: "2ea3b37a987ea551b3eb93a9238158dd"
server_key_version: "encryptionkey@0"
I20260928 23:52:17.149946 21585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:17.163034 21585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:17.165761 21585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:17.165962 21585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:17.172477 21585 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:38395
I20260928 23:52:17.172761 21652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:38395 every 8 connection(s)
I20260928 23:52:17.173714 21585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:52:17.177701 21653 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:52:17.179250 21653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3: Bootstrap starting.
I20260928 23:52:17.183245 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 21585
I20260928 23:52:17.183691 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:36277
--local_ip_for_outbound_sockets=127.6.152.1
--tserver_master_addrs=127.6.152.62:38395
--webserver_port=37833
--webserver_interface=127.6.152.1
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:52:17.196175 21653 log.cc:824] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:17.206881 21653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:52:17.207243 21653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3: Bootstrap complete.
I20260928 23:52:17.211164 21653 raft_consensus.cc:348] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04ec9210a964749a041cf9a3d387ae3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38395 } }
I20260928 23:52:17.211558 21653 raft_consensus.cc:729] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d04ec9210a964749a041cf9a3d387ae3, State: Initialized, Role: FOLLOWER
I20260928 23:52:17.212064 21653 consensus_queue.cc:260] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04ec9210a964749a041cf9a3d387ae3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38395 } }
I20260928 23:52:17.212206 21653 raft_consensus.cc:388] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:52:17.212301 21653 raft_consensus.cc:482] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:52:17.212412 21653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [term 2 FOLLOWER]: Advancing to term 3
I20260928 23:52:17.214231 21653 raft_consensus.cc:504] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04ec9210a964749a041cf9a3d387ae3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38395 } }
I20260928 23:52:17.214556 21653 leader_election.cc:302] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d04ec9210a964749a041cf9a3d387ae3; no voters:
I20260928 23:52:17.214888 21653 leader_election.cc:288] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 23:52:17.214970 21657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 23:52:17.215153 21657 raft_consensus.cc:686] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [term 3 LEADER]: Becoming Leader. State: Replica: d04ec9210a964749a041cf9a3d387ae3, State: Running, Role: LEADER
I20260928 23:52:17.215440 21657 consensus_queue.cc:237] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04ec9210a964749a041cf9a3d387ae3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38395 } }
I20260928 23:52:17.216030 21653 sys_catalog.cc:564] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:52:17.216354 21659 sys_catalog.cc:455] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d04ec9210a964749a041cf9a3d387ae3. Latest consensus state: current_term: 3 leader_uuid: "d04ec9210a964749a041cf9a3d387ae3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04ec9210a964749a041cf9a3d387ae3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38395 } } }
I20260928 23:52:17.216476 21659 sys_catalog.cc:458] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [sys.catalog]: This master's current role is: LEADER
I20260928 23:52:17.216708 21658 sys_catalog.cc:455] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d04ec9210a964749a041cf9a3d387ae3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d04ec9210a964749a041cf9a3d387ae3" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38395 } } }
I20260928 23:52:17.216868 21658 sys_catalog.cc:458] T 00000000000000000000000000000000 P d04ec9210a964749a041cf9a3d387ae3 [sys.catalog]: This master's current role is: LEADER
I20260928 23:52:17.217312 21661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:52:17.222134 21661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:52:17.222832 21661 catalog_manager.cc:1259] Loaded cluster ID: 3ab7fb17a3d446bdb33ee47962501e5d
I20260928 23:52:17.222898 21661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 23:52:17.223677 21661 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1f4a2009ad google::LogMessage::Fail() at ??:0
@ 0x7f1f4a20388a google::LogMessage::SendToLog() at ??:0
@ 0x7f1f4a2004de google::LogMessage::Flush() at ??:0
@ 0x7f1f4a201a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1f54b3ffdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f1f54b40d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1f54b3d579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f1f54b81640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1f4c4523d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1f4c452c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f1f4c4543f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1f4c443512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1f4da876db start_thread at ??:0
@ 0x7f1f48cfe61f clone at ??:0
W20260928 23:52:17.327312 21655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:17.327662 21655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:17.327728 21655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:17.333912 21655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:17.333993 21655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:17.334082 21655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:52:17.342182 21655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:36277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=37833
--tserver_master_addrs=127.6.152.62:38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:17.343629 21655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:17.345024 21655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:17.355423 21694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:17.355429 21697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:17.355465 21695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:17.355706 21655 server_base.cc:1034] running on GCE node
I20260928 23:52:17.356101 21655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:17.356747 21655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:17.357954 21655 hybrid_clock.cc:648] HybridClock initialized: now 1790639537357941 us; error 38 us; skew 500 ppm
Sep 28 23:52:17 dist-test-slave-p5bw krb5kdc[20509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639537, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:17.360598 21655 init.cc:377] Logged in from keytab as kudu/127.6.152.1@KRBTEST.COM (short username kudu)
I20260928 23:52:17.361529 21655 webserver.cc:460] Webserver started at http://127.6.152.1:37833/ using document root <none> and password file <none>
I20260928 23:52:17.362085 21655 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:17.362145 21655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:17.365998 21655 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:52:17.368184 21704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:17.369459 21655 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:52:17.369593 21655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "3ddfefce2d894f6697bc7011773750bf"
format_stamp: "Formatted at 2026-09-28 23:52:15 on dist-test-slave-p5bw"
server_key: "3aedbeb410904fbcba587d29f530446f"
server_key_iv: "28fe90867f765bd5a57dfdcafa757337"
server_key_version: "encryptionkey@0"
I20260928 23:52:17.370137 21655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:17.383808 21655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:17.386885 21655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:17.387085 21655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:17.387696 21655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:52:17.388857 21711 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 23:52:17.392846 21655 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 23:52:17.392916 21655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260928 23:52:17.392985 21655 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 23:52:17.394672 21655 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 23:52:17.394731 21655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 23:52:17.394980 21711 tablet_bootstrap.cc:492] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf: Bootstrap starting.
I20260928 23:52:17.405229 21655 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:36277
I20260928 23:52:17.405488 21818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:36277 every 8 connection(s)
I20260928 23:52:17.406507 21655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:52:17.410292 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 21655
I20260928 23:52:17.410774 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:43285
--local_ip_for_outbound_sockets=127.6.152.2
--tserver_master_addrs=127.6.152.62:38395
--webserver_port=35505
--webserver_interface=127.6.152.2
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:52:17.415758 21711 log.cc:824] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:17.422689 21711 tablet_bootstrap.cc:492] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:52:17.423066 21711 tablet_bootstrap.cc:492] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf: Bootstrap complete.
I20260928 23:52:17.424156 21711 ts_tablet_manager.cc:1397] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf: Time spent bootstrapping tablet: real 0.029s user 0.022s sys 0.004s
I20260928 23:52:17.426604 21711 raft_consensus.cc:348] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } }
I20260928 23:52:17.427032 21711 raft_consensus.cc:729] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ddfefce2d894f6697bc7011773750bf, State: Initialized, Role: FOLLOWER
I20260928 23:52:17.427512 21711 consensus_queue.cc:260] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } }
I20260928 23:52:17.428431 21711 ts_tablet_manager.cc:1428] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
W20260928 23:52:17.454855 21821 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:52:17.408889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.1:48155)
0928 23:52:17.409451 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:17.409511 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:17.410633 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:17.453914 (+ 43281us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":156,"threads_started":1}
W20260928 23:52:17.455462 21819 heartbeater.cc:644] Failed to heartbeat to 127.6.152.62:38395 (0 consecutive failures): Network error: Failed to ping master at 127.6.152.62:38395: Client connection negotiation failed: client connection to 127.6.152.62:38395: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 23:52:17.456190 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:17.455915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.1:54883)
0928 23:52:17.456050 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:17.456125 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 23:52:17.456861 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:17.456627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.1:53227)
0928 23:52:17.456737 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:17.456788 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 23:52:17.457048 21819 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:52:17.534126 21822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:17.534358 21822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:17.534402 21822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:17.538372 21822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:17.538420 21822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:17.538516 21822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:52:17.543663 21822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:43285
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=35505
--tserver_master_addrs=127.6.152.62:38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:17.544642 21822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:17.545627 21822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:17.554890 21829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:17.554948 21822 server_base.cc:1034] running on GCE node
W20260928 23:52:17.554919 21830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:17.554890 21832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:17.555452 21822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:17.556115 21822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:17.557301 21822 hybrid_clock.cc:648] HybridClock initialized: now 1790639537557283 us; error 29 us; skew 500 ppm
Sep 28 23:52:17 dist-test-slave-p5bw krb5kdc[20509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639537, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:17.566920 21822 init.cc:377] Logged in from keytab as kudu/127.6.152.2@KRBTEST.COM (short username kudu)
I20260928 23:52:17.567893 21822 webserver.cc:460] Webserver started at http://127.6.152.2:35505/ using document root <none> and password file <none>
I20260928 23:52:17.568465 21822 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:17.568555 21822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:17.572471 21822 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:52:17.574867 21839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:17.576229 21822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 23:52:17.576375 21822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "5800969806274705b9f893edd0f6a069"
format_stamp: "Formatted at 2026-09-28 23:52:15 on dist-test-slave-p5bw"
server_key: "1f45295d76092946fe0cd5ea6b7a8a9d"
server_key_iv: "b444aa4e08a90c67bcf9b8b1ac81b01a"
server_key_version: "encryptionkey@0"
I20260928 23:52:17.576956 21822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:17.594688 21822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:17.597935 21822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:17.598145 21822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:17.598737 21822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:52:17.599538 21822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:52:17.599607 21822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:17.599681 21822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:52:17.599725 21822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:17.608492 21822 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:43285
I20260928 23:52:17.608513 21952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:43285 every 8 connection(s)
I20260928 23:52:17.609519 21822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:52:17.610505 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 21822
I20260928 23:52:17.610911 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:43391
--local_ip_for_outbound_sockets=127.6.152.3
--tserver_master_addrs=127.6.152.62:38395
--webserver_port=35319
--webserver_interface=127.6.152.3
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:52:17.613585 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:17.611757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.2:42041)
0928 23:52:17.612277 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:17.612390 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":156,"threads_started":1}
W20260928 23:52:17.614007 21953 heartbeater.cc:644] Failed to heartbeat to 127.6.152.62:38395 (0 consecutive failures): Network error: Failed to ping master at 127.6.152.62:38395: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
I20260928 23:52:17.614631 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:17.614397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.2:52975)
0928 23:52:17.614514 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:17.614556 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 23:52:17.615273 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:17.615105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.2:48729)
0928 23:52:17.615194 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:17.615231 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 23:52:17.615482 21953 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:52:17.735360 21955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:17.735642 21955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:17.735718 21955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:17.740020 21955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:17.740127 21955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:17.740226 21955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:52:17.745770 21955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:43391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=35319
--tserver_master_addrs=127.6.152.62:38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:17.746783 21955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:17.747772 21955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:17.757019 21961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:17.757025 21962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:17.757165 21964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:17.757074 21955 server_base.cc:1034] running on GCE node
I20260928 23:52:17.757587 21955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:17.758221 21955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:17.759404 21955 hybrid_clock.cc:648] HybridClock initialized: now 1790639537759371 us; error 45 us; skew 500 ppm
Sep 28 23:52:17 dist-test-slave-p5bw krb5kdc[20509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639537, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:17.761991 21955 init.cc:377] Logged in from keytab as kudu/127.6.152.3@KRBTEST.COM (short username kudu)
I20260928 23:52:17.762822 21955 webserver.cc:460] Webserver started at http://127.6.152.3:35319/ using document root <none> and password file <none>
I20260928 23:52:17.763329 21955 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:17.763388 21955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:17.766968 21955 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:52:17.769023 21971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:17.770226 21955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:52:17.770349 21955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "ed0f7ba578aa4ba2b5132c9f4741190f"
format_stamp: "Formatted at 2026-09-28 23:52:15 on dist-test-slave-p5bw"
server_key: "4e0d9997cbb1330a432433630f406c41"
server_key_iv: "6eb8620012a4a37648682c2e55361d3f"
server_key_version: "encryptionkey@0"
I20260928 23:52:17.770860 21955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:17.788022 21955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:17.790674 21955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:17.790870 21955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:17.791414 21955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:52:17.792183 21955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:52:17.792227 21955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:17.792294 21955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:52:17.792336 21955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:17.801043 21955 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:43391
I20260928 23:52:17.801088 22084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:43391 every 8 connection(s)
I20260928 23:52:17.801977 21955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:52:17.805555 22087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:17.803883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.3:43247)
0928 23:52:17.804299 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:17.804400 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":110,"threads_started":1}
W20260928 23:52:17.805989 22085 heartbeater.cc:644] Failed to heartbeat to 127.6.152.62:38395 (0 consecutive failures): Network error: Failed to ping master at 127.6.152.62:38395: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
I20260928 23:52:17.806596 22087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:17.806383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.3:33553)
0928 23:52:17.806492 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:17.806546 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 23:52:17.807193 22087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:17.806991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.3:58863)
0928 23:52:17.807094 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:17.807152 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 23:52:17.807371 22085 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 23:52:17.807785 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 21955
I20260928 23:52:18.458141 22089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:18.457637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.1:54721)
0928 23:52:18.457947 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:18.458007 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":84,"threads_started":1}
I20260928 23:52:18.616444 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:18.616022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.2:55393)
0928 23:52:18.616288 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:18.616344 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":70,"threads_started":1}
I20260928 23:52:18.808248 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:18.807888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.3:54767)
0928 23:52:18.808108 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:18.808158 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":65,"threads_started":1}
I20260928 23:52:19.145678 22092 raft_consensus.cc:482] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:52:19.145900 22092 raft_consensus.cc:504] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } }
I20260928 23:52:19.147017 22092 leader_election.cc:288] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed0f7ba578aa4ba2b5132c9f4741190f (127.6.152.3:43391), 5800969806274705b9f893edd0f6a069 (127.6.152.2:43285)
Sep 28 23:52:19 dist-test-slave-p5bw krb5kdc[20509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639537, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for kudu/127.6.152.2@KRBTEST.COM
Sep 28 23:52:19 dist-test-slave-p5bw krb5kdc[20509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639537, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for kudu/127.6.152.3@KRBTEST.COM
I20260928 23:52:19.161160 22094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:19.147283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:43285 (local address 127.6.152.1:44947)
0928 23:52:19.147623 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:19.147639 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:19.147759 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:19.149383 (+ 1624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:19.149389 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:19.149910 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:19.150396 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:19.150410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:19.151676 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:19.151681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:19.152923 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:19.152933 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:19.153731 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:19.153736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:19.153921 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:19.154656 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:19.154669 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:19.156315 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:19.158722 (+ 2407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:19.158728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:19.158742 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:19.159057 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:19.159409 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:19.159413 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:19.159415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:19.159556 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:19.160172 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:19.160178 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:19.160401 (+ 223us) client_negotiation.cc:769] Sending connection context
0928 23:52:19.160585 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 23:52:19.160959 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":131,"spinlock_wait_cycles":29440,"thread_start_us":76,"threads_started":1}
I20260928 23:52:19.161172 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:19.147467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:44947 (local address 127.6.152.2:43285)
0928 23:52:19.148012 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:19.148022 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:19.148056 (+ 34us) server_negotiation.cc:407] Connection header received
0928 23:52:19.148683 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:19.148689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:19.149044 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:19.149331 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:19.150575 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:19.151528 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:19.153043 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:19.153578 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:19.153759 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:19.156575 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:19.156580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:19.156592 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:19.156621 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:19.158582 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:19.159176 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:19.159181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:19.159187 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:19.159311 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:19.159652 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:19.159657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:19.159658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:19.160047 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:19.160188 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:19.160701 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 23:52:19.160981 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":70,"threads_started":1}
I20260928 23:52:19.161198 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:19.147467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:41875 (local address 127.6.152.3:43391)
0928 23:52:19.147858 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:19.147880 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:19.147917 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:52:19.148579 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:19.148584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:19.149005 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:19.149331 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:19.150568 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:19.151532 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:19.153044 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:19.153627 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:19.153853 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:19.156916 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:19.156921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:19.156933 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:19.156968 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:19.158629 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:19.159166 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:19.159180 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:19.159185 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:19.159307 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:19.159643 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:19.159648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:19.159650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:19.160047 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:19.160188 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:19.160700 (+ 512us) server_negotiation.cc:299] Negotiation successful
0928 23:52:19.161008 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":65,"threads_started":1}
I20260928 23:52:19.161211 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:19.147283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:43391 (local address 127.6.152.1:41875)
0928 23:52:19.147725 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:19.147737 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:19.147802 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:19.149383 (+ 1581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:19.149389 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:19.149910 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:19.150396 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:19.150412 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:19.151685 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:19.151688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:19.152925 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:19.152935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:19.153820 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:19.153824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:19.153938 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:19.154591 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:19.154607 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:19.156691 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:19.158750 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:19.158754 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:19.158758 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:19.159047 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:19.159405 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:19.159409 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:19.159411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:19.159550 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:19.160383 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:19.160385 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:19.160458 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 23:52:19.160585 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 23:52:19.161040 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":382,"thread_start_us":50,"threads_started":1}
W20260928 23:52:19.164419 21705 leader_election.cc:341] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed0f7ba578aa4ba2b5132c9f4741190f (127.6.152.3:43391): Not found: Tablet not found: c96714228ad043e18499097af9fe1257
W20260928 23:52:19.164633 21708 leader_election.cc:341] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5800969806274705b9f893edd0f6a069 (127.6.152.2:43285): Not found: Tablet not found: c96714228ad043e18499097af9fe1257
I20260928 23:52:19.164710 21708 leader_election.cc:302] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ddfefce2d894f6697bc7011773750bf; no voters: 5800969806274705b9f893edd0f6a069, ed0f7ba578aa4ba2b5132c9f4741190f
I20260928 23:52:19.164971 22092 raft_consensus.cc:2726] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:52:19.459136 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:19.458865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.1:36137)
0928 23:52:19.458995 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:19.459057 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 23:52:19.617589 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:19.617201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.2:57855)
0928 23:52:19.617435 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:19.617490 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":76,"threads_started":1}
I20260928 23:52:19.809319 22098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:19.808965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.3:53649)
0928 23:52:19.809185 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:19.809237 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":66,"threads_started":1}
I20260928 23:52:20.460234 22099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:20.459845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.1:58161)
0928 23:52:20.460091 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:20.460144 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
I20260928 23:52:20.618718 22100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:20.618315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.2:47389)
0928 23:52:20.618568 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:20.618626 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
I20260928 23:52:20.785465 22101 raft_consensus.cc:482] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:52:20.785590 22101 raft_consensus.cc:504] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } }
I20260928 23:52:20.786139 22101 leader_election.cc:288] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed0f7ba578aa4ba2b5132c9f4741190f (127.6.152.3:43391), 5800969806274705b9f893edd0f6a069 (127.6.152.2:43285)
W20260928 23:52:20.787075 21708 leader_election.cc:341] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5800969806274705b9f893edd0f6a069 (127.6.152.2:43285): Not found: Tablet not found: c96714228ad043e18499097af9fe1257
W20260928 23:52:20.787204 21705 leader_election.cc:341] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed0f7ba578aa4ba2b5132c9f4741190f (127.6.152.3:43391): Not found: Tablet not found: c96714228ad043e18499097af9fe1257
I20260928 23:52:20.787292 21705 leader_election.cc:302] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ddfefce2d894f6697bc7011773750bf; no voters: 5800969806274705b9f893edd0f6a069, ed0f7ba578aa4ba2b5132c9f4741190f
I20260928 23:52:20.787420 22101 raft_consensus.cc:2726] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:52:20.810461 22102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:20.810042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.3:49747)
0928 23:52:20.810303 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:20.810362 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":101,"threads_started":1}
I20260928 23:52:21.461602 22103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:21.461151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.1:56419)
0928 23:52:21.461439 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:21.461506 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":81,"threads_started":1}
I20260928 23:52:21.619868 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:21.619475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.2:60967)
0928 23:52:21.619704 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:21.619760 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":70,"threads_started":1}
I20260928 23:52:21.811664 22105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:21.811230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.3:38199)
0928 23:52:21.811494 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:21.811558 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":74,"threads_started":1}
I20260928 23:52:22.462949 22106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:22.462488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.1:53635)
0928 23:52:22.462798 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:22.462855 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":73,"threads_started":1}
I20260928 23:52:22.621223 22107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:22.620744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.2:41585)
0928 23:52:22.621070 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:22.621129 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":141,"threads_started":1}
I20260928 23:52:22.813009 22108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:22.812517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.3:39799)
0928 23:52:22.812798 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:22.812913 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":70,"threads_started":1}
I20260928 23:52:22.980343 22109 raft_consensus.cc:482] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:52:22.980537 22109 raft_consensus.cc:504] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } }
I20260928 23:52:22.981055 22109 leader_election.cc:288] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed0f7ba578aa4ba2b5132c9f4741190f (127.6.152.3:43391), 5800969806274705b9f893edd0f6a069 (127.6.152.2:43285)
W20260928 23:52:22.981789 21708 leader_election.cc:341] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5800969806274705b9f893edd0f6a069 (127.6.152.2:43285): Not found: Tablet not found: c96714228ad043e18499097af9fe1257
W20260928 23:52:22.981941 21705 leader_election.cc:341] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed0f7ba578aa4ba2b5132c9f4741190f (127.6.152.3:43391): Not found: Tablet not found: c96714228ad043e18499097af9fe1257
I20260928 23:52:22.982043 21705 leader_election.cc:302] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ddfefce2d894f6697bc7011773750bf; no voters: 5800969806274705b9f893edd0f6a069, ed0f7ba578aa4ba2b5132c9f4741190f
I20260928 23:52:22.982239 22109 raft_consensus.cc:2726] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:52:23.464080 22110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:23.463670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.1:37077)
0928 23:52:23.463935 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:23.463989 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":73,"threads_started":1}
I20260928 23:52:23.622539 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:23.622129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.2:55961)
0928 23:52:23.622379 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:23.622439 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
I20260928 23:52:23.814090 22112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:23.813723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.3:53887)
0928 23:52:23.813949 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:23.814000 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":71,"threads_started":1}
I20260928 23:52:24.465242 22113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:24.464849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.1:41579)
0928 23:52:24.465092 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:24.465146 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":69,"threads_started":1}
I20260928 23:52:24.623732 22114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:24.623311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.2:43965)
0928 23:52:24.623587 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:24.623637 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":70,"threads_started":1}
I20260928 23:52:24.815230 22115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:24.814805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.3:41945)
0928 23:52:24.815089 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:24.815144 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":79,"threads_started":1}
I20260928 23:52:25.466514 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:25.466013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.1:47797)
0928 23:52:25.466350 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:25.466419 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":103,"threads_started":1}
I20260928 23:52:25.624949 22117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:25.624484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.2:40925)
0928 23:52:25.624772 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:25.624853 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":89,"threads_started":1}
I20260928 23:52:25.816470 22118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:25.816092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.3:57739)
0928 23:52:25.816326 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:25.816379 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":74,"threads_started":1}
I20260928 23:52:26.467638 22119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:26.467235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.1:53165)
0928 23:52:26.467484 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:26.467541 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":73,"threads_started":1}
I20260928 23:52:26.626092 22120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:26.625707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.2:38373)
0928 23:52:26.625942 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:26.625996 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":71,"threads_started":1}
I20260928 23:52:26.817626 22121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:26.817246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.3:55639)
0928 23:52:26.817485 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:26.817539 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":77,"threads_started":1}
I20260928 23:52:27.469046 22122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:27.468621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.1:45119)
0928 23:52:27.468897 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:27.468951 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":76,"threads_started":1}
I20260928 23:52:27.529883 22123 raft_consensus.cc:482] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:52:27.530021 22123 raft_consensus.cc:504] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } }
I20260928 23:52:27.530543 22123 leader_election.cc:288] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed0f7ba578aa4ba2b5132c9f4741190f (127.6.152.3:43391), 5800969806274705b9f893edd0f6a069 (127.6.152.2:43285)
W20260928 23:52:27.531422 21708 leader_election.cc:341] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5800969806274705b9f893edd0f6a069 (127.6.152.2:43285): Not found: Tablet not found: c96714228ad043e18499097af9fe1257
W20260928 23:52:27.531515 21705 leader_election.cc:341] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed0f7ba578aa4ba2b5132c9f4741190f (127.6.152.3:43391): Not found: Tablet not found: c96714228ad043e18499097af9fe1257
I20260928 23:52:27.531591 21705 leader_election.cc:302] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ddfefce2d894f6697bc7011773750bf; no voters: 5800969806274705b9f893edd0f6a069, ed0f7ba578aa4ba2b5132c9f4741190f
I20260928 23:52:27.531818 22123 raft_consensus.cc:2726] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:52:27.627327 22124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:27.626920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.2:52755)
0928 23:52:27.627159 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:27.627215 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
I20260928 23:52:27.818974 22125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:27.818519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.3:46167)
0928 23:52:27.818792 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:27.818877 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":83,"threads_started":1}
I20260928 23:52:28.470268 22126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:28.469869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.1:59409)
0928 23:52:28.470111 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:28.470171 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":74,"threads_started":1}
I20260928 23:52:28.628525 22127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:28.628048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.2:52303)
0928 23:52:28.628332 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:28.628410 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":73,"threads_started":1}
I20260928 23:52:28.820202 22128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:28.819794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.3:51003)
0928 23:52:28.820040 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:28.820100 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":70,"threads_started":1}
I20260928 23:52:29.471582 22129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:29.471086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.1:33489)
0928 23:52:29.471430 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:29.471488 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":70,"threads_started":1}
I20260928 23:52:29.629630 22130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:29.629227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.2:46819)
0928 23:52:29.629479 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:29.629538 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":96,"threads_started":1}
I20260928 23:52:29.821600 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:29.821168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.3:36707)
0928 23:52:29.821441 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:29.821502 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":85,"threads_started":1}
I20260928 23:52:30.472878 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:30.472360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.1:51285)
0928 23:52:30.472674 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:30.472731 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":110,"threads_started":1}
I20260928 23:52:30.630854 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:30.630447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.2:51731)
0928 23:52:30.630695 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:30.630757 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":91,"threads_started":1}
I20260928 23:52:30.822863 22134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:30.822426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.3:59115)
0928 23:52:30.822694 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:30.822770 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":90,"threads_started":1}
I20260928 23:52:31.474042 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:31.473620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.1:43613)
0928 23:52:31.473890 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:31.473945 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":93,"threads_started":1}
I20260928 23:52:31.631989 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:31.631575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.2:46215)
0928 23:52:31.631842 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:31.631896 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":86,"threads_started":1}
I20260928 23:52:31.824024 22137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:31.823614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.3:42919)
0928 23:52:31.823875 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:31.823930 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":74,"threads_started":1}
I20260928 23:52:32.475431 22138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:32.475021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.1:54537)
0928 23:52:32.475268 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:32.475325 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
I20260928 23:52:32.488471 22139 raft_consensus.cc:482] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:52:32.488595 22139 raft_consensus.cc:504] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0f7ba578aa4ba2b5132c9f4741190f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 43391 } } peers { permanent_uuid: "5800969806274705b9f893edd0f6a069" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 43285 } } peers { permanent_uuid: "3ddfefce2d894f6697bc7011773750bf" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36277 } }
I20260928 23:52:32.489086 22139 leader_election.cc:288] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed0f7ba578aa4ba2b5132c9f4741190f (127.6.152.3:43391), 5800969806274705b9f893edd0f6a069 (127.6.152.2:43285)
W20260928 23:52:32.489840 21708 leader_election.cc:341] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5800969806274705b9f893edd0f6a069 (127.6.152.2:43285): Not found: Tablet not found: c96714228ad043e18499097af9fe1257
W20260928 23:52:32.489944 21705 leader_election.cc:341] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed0f7ba578aa4ba2b5132c9f4741190f (127.6.152.3:43391): Not found: Tablet not found: c96714228ad043e18499097af9fe1257
I20260928 23:52:32.490002 21705 leader_election.cc:302] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ddfefce2d894f6697bc7011773750bf; no voters: 5800969806274705b9f893edd0f6a069, ed0f7ba578aa4ba2b5132c9f4741190f
I20260928 23:52:32.490142 22139 raft_consensus.cc:2726] T c96714228ad043e18499097af9fe1257 P 3ddfefce2d894f6697bc7011773750bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:52:32.633167 22140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:32.632752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.2:44529)
0928 23:52:32.633024 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:32.633073 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":121,"threads_started":1}
I20260928 23:52:32.813835 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 21655
I20260928 23:52:32.819806 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 21822
I20260928 23:52:32.825416 22141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:32.824970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38395 (local address 127.6.152.3:40651)
0928 23:52:32.825264 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:32.825321 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":83,"threads_started":1}
I20260928 23:52:32.825578 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 21955
2026-09-28T23:52:32Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19225 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:52:32 dist-test-slave-p5bw krb5kdc[22144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:52:32 dist-test-slave-p5bw krb5kdc[22144](info): set up 2 sockets
Sep 28 23:52:32 dist-test-slave-p5bw krb5kdc[22144](info): commencing operation
krb5kdc: starting...
W20260928 23:52:33.869704 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:52:33 dist-test-slave-p5bw krb5kdc[22144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:52:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:52:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:52:33.997490 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:45073
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:45073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:52:34.107280 22157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:34.107585 22157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:34.107659 22157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:34.111642 22157 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 23:52:34.111724 22157 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 23:52:34.111752 22157 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 23:52:34.111794 22157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:52:34.111862 22157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:34.111901 22157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:34.111936 22157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:52:34.111971 22157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:52:34.117318 22157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:45073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:45073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:34.118458 22157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:34.119473 22157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:34.128283 22165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:34.128283 22162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:34.128381 22157 server_base.cc:1034] running on GCE node
W20260928 23:52:34.128247 22163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:34.129181 22157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:34.130255 22157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:34.131481 22157 hybrid_clock.cc:648] HybridClock initialized: now 1790639554131449 us; error 42 us; skew 500 ppm
Sep 28 23:52:34 dist-test-slave-p5bw krb5kdc[22144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639554, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:34.134332 22157 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:52:34.135208 22157 webserver.cc:460] Webserver started at http://127.6.152.62:40941/ using document root <none> and password file <none>
I20260928 23:52:34.135772 22157 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:34.135862 22157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:34.136089 22157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:52:34.138167 22157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "0ab3dc9959304dbb9b902fc81ace7f43"
format_stamp: "Formatted at 2026-09-28 23:52:34 on dist-test-slave-p5bw"
server_key: "466ede8102a6d02b8086f4f74b07eefc"
server_key_iv: "c1df7aba165e5e63b5d313373f6009d7"
server_key_version: "encryptionkey@0"
I20260928 23:52:34.138742 22157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "0ab3dc9959304dbb9b902fc81ace7f43"
format_stamp: "Formatted at 2026-09-28 23:52:34 on dist-test-slave-p5bw"
server_key: "466ede8102a6d02b8086f4f74b07eefc"
server_key_iv: "c1df7aba165e5e63b5d313373f6009d7"
server_key_version: "encryptionkey@0"
I20260928 23:52:34.142467 22157 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 23:52:34.144804 22172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:34.146119 22157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:52:34.146310 22157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "0ab3dc9959304dbb9b902fc81ace7f43"
format_stamp: "Formatted at 2026-09-28 23:52:34 on dist-test-slave-p5bw"
server_key: "466ede8102a6d02b8086f4f74b07eefc"
server_key_iv: "c1df7aba165e5e63b5d313373f6009d7"
server_key_version: "encryptionkey@0"
I20260928 23:52:34.146423 22157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:34.159204 22157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:34.162141 22157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:34.162364 22157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:34.169263 22157 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:45073
I20260928 23:52:34.169574 22224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:45073 every 8 connection(s)
I20260928 23:52:34.170225 22157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:52:34.172909 22225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:52:34.173425 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 22157
I20260928 23:52:34.173534 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:52:34.173892 6752 external_mini_cluster.cc:1428] Setting key 6c44f4ab288cfa01aaacdedd612dc4d6
I20260928 23:52:34.179915 22225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43: Bootstrap starting.
I20260928 23:52:34.182653 22225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43: Neither blocks nor log segments found. Creating new log.
Sep 28 23:52:34 dist-test-slave-p5bw krb5kdc[22144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:52:34.183497 22225 log.cc:824] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:34.185770 22225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43: No bootstrap required, opened a new log
I20260928 23:52:34.188446 22228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:34.175243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59592 (local address 127.6.152.62:45073)
0928 23:52:34.175643 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:34.175657 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:34.175691 (+ 34us) server_negotiation.cc:407] Connection header received
0928 23:52:34.176376 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:34.176393 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:34.176704 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:34.177003 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:34.177868 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.178857 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.179981 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.180488 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.180603 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:34.183441 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:34.183447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:34.183464 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:34.183493 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:34.185421 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:34.185869 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:34.185875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:34.185880 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:34.186010 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:34.186281 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:34.186284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:34.186286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:34.186764 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:34.186957 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:34.187147 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 23:52:34.187431 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":131,"threads_started":1}
I20260928 23:52:34.190145 22225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab3dc9959304dbb9b902fc81ace7f43" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45073 } }
I20260928 23:52:34.190414 22225 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:52:34.190461 22225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab3dc9959304dbb9b902fc81ace7f43, State: Initialized, Role: FOLLOWER
I20260928 23:52:34.190907 22225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab3dc9959304dbb9b902fc81ace7f43" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45073 } }
I20260928 23:52:34.191028 22225 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:52:34.191068 22225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:52:34.191135 22225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:52:34.191928 22225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab3dc9959304dbb9b902fc81ace7f43" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45073 } }
I20260928 23:52:34.192201 22225 leader_election.cc:302] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ab3dc9959304dbb9b902fc81ace7f43; no voters:
I20260928 23:52:34.192435 22225 leader_election.cc:288] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:52:34.192559 22230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:52:34.192849 22230 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [term 1 LEADER]: Becoming Leader. State: Replica: 0ab3dc9959304dbb9b902fc81ace7f43, State: Running, Role: LEADER
I20260928 23:52:34.193456 22230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab3dc9959304dbb9b902fc81ace7f43" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45073 } }
I20260928 23:52:34.193740 22225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:52:34.195040 22232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ab3dc9959304dbb9b902fc81ace7f43. Latest consensus state: current_term: 1 leader_uuid: "0ab3dc9959304dbb9b902fc81ace7f43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab3dc9959304dbb9b902fc81ace7f43" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45073 } } }
I20260928 23:52:34.195092 22231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ab3dc9959304dbb9b902fc81ace7f43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab3dc9959304dbb9b902fc81ace7f43" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45073 } } }
I20260928 23:52:34.195163 22232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [sys.catalog]: This master's current role is: LEADER
I20260928 23:52:34.195200 22231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [sys.catalog]: This master's current role is: LEADER
I20260928 23:52:34.195575 22241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:52:34.198369 22241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:52:34.203346 22241 catalog_manager.cc:1347] Generated new cluster ID: 5a5b082fa963486ea4bcf15a596c8950
I20260928 23:52:34.203410 22241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:52:34.224645 22241 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:52:34.225695 22241 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:52:34.231142 22241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43: Generated new TSK 0
I20260928 23:52:34.231797 22241 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
I20260928 23:52:34.265337 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:45073
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:52:34.375873 22251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:34.376123 22251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:34.376164 22251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:34.380115 22251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:34.380172 22251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:34.380338 22251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:52:34.385401 22251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:45073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:34.386319 22251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:34.387315 22251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:34.397130 22259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:34.397117 22257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:34.397105 22256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:34.397173 22251 server_base.cc:1034] running on GCE node
I20260928 23:52:34.397764 22251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:34.398401 22251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:34.399591 22251 hybrid_clock.cc:648] HybridClock initialized: now 1790639554399579 us; error 26 us; skew 500 ppm
Sep 28 23:52:34 dist-test-slave-p5bw krb5kdc[22144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639554, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:34.402411 22251 init.cc:377] Logged in from keytab as kudu/127.6.152.1@KRBTEST.COM (short username kudu)
I20260928 23:52:34.403205 22251 webserver.cc:460] Webserver started at http://127.6.152.1:34805/ using document root <none> and password file <none>
I20260928 23:52:34.403695 22251 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:34.403752 22251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:34.403908 22251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:52:34.405853 22251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "47fc0f3fa1e649e8ac43fb560b6fd753"
format_stamp: "Formatted at 2026-09-28 23:52:34 on dist-test-slave-p5bw"
server_key: "7b1a85b8d117e3a44d10cccb9f0eb552"
server_key_iv: "1b7c7dc9a6b806fcd3ec615956a36197"
server_key_version: "encryptionkey@0"
I20260928 23:52:34.406350 22251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "47fc0f3fa1e649e8ac43fb560b6fd753"
format_stamp: "Formatted at 2026-09-28 23:52:34 on dist-test-slave-p5bw"
server_key: "7b1a85b8d117e3a44d10cccb9f0eb552"
server_key_iv: "1b7c7dc9a6b806fcd3ec615956a36197"
server_key_version: "encryptionkey@0"
I20260928 23:52:34.409799 22251 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:52:34.412071 22266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:34.413493 22251 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:52:34.413633 22251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "47fc0f3fa1e649e8ac43fb560b6fd753"
format_stamp: "Formatted at 2026-09-28 23:52:34 on dist-test-slave-p5bw"
server_key: "7b1a85b8d117e3a44d10cccb9f0eb552"
server_key_iv: "1b7c7dc9a6b806fcd3ec615956a36197"
server_key_version: "encryptionkey@0"
I20260928 23:52:34.413750 22251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:34.438426 22251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:34.441861 22251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:34.442070 22251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:34.442675 22251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:52:34.443504 22251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:52:34.443574 22251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:34.443645 22251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:52:34.443687 22251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:34.452257 22251 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:33381
I20260928 23:52:34.452318 22379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:33381 every 8 connection(s)
I20260928 23:52:34.453209 22251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:52:34.461776 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 22251
I20260928 23:52:34.461898 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
Sep 28 23:52:34 dist-test-slave-p5bw krb5kdc[22144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639554, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:52:34.462221 6752 external_mini_cluster.cc:1428] Setting key 5130af92fb3dc98e673ae6e1b5249f78
I20260928 23:52:34.466638 22228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:34.454922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:52745 (local address 127.6.152.62:45073)
0928 23:52:34.455058 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:34.455061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:34.455872 (+ 811us) server_negotiation.cc:407] Connection header received
0928 23:52:34.457151 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:34.457153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:34.457199 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:34.457273 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:34.458370 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.458889 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.459954 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.460170 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.460243 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:34.462675 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:34.462678 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:34.462680 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:34.462702 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:34.464476 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:34.465077 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:34.465083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:34.465088 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:34.465179 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:34.465528 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:34.465531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:34.465533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:34.465727 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:34.465834 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:34.466342 (+ 508us) server_negotiation.cc:299] Negotiation successful
0928 23:52:34.466460 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:52:34.467434 22382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:34.455292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.1:52745)
0928 23:52:34.455721 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:34.455762 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:34.456968 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:34.457427 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:34.457433 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:34.457771 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:34.458241 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:34.458254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.458992 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.458994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:34.459853 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:34.459862 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.460289 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.460292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:34.460422 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:34.460992 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:34.461007 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:34.462492 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:34.464602 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:34.464607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:34.464619 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:34.464979 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:34.465282 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:34.465284 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:34.465286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:34.465414 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:34.465842 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:34.465850 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:34.466076 (+ 226us) client_negotiation.cc:769] Sending connection context
0928 23:52:34.466284 (+ 208us) client_negotiation.cc:240] Negotiation successful
0928 23:52:34.466591 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":120,"threads_started":1}
I20260928 23:52:34.468628 22380 heartbeater.cc:347] Connected to a master server at 127.6.152.62:45073
I20260928 23:52:34.468925 22380 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:34.469398 22380 heartbeater.cc:511] Master 127.6.152.62:45073 requested a full tablet report, sending...
I20260928 23:52:34.470827 22189 ts_manager.cc:194] Registered new tserver with Master: 47fc0f3fa1e649e8ac43fb560b6fd753 (127.6.152.1:33381)
WARNING: no policy specified for kudu/127.6.152.2@KRBTEST.COM; defaulting to no policy
I20260928 23:52:34.471990 22189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.1@KRBTEST.COM'} at 127.6.152.1:52745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
I20260928 23:52:34.489537 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:45073
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:52:34.598227 22385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:34.598480 22385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:34.598523 22385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:34.602434 22385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:34.602487 22385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:34.602545 22385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:52:34.607863 22385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:45073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:34.608866 22385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:34.609818 22385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:34.619268 22390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:34.619263 22391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:34.619264 22393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:34.619539 22385 server_base.cc:1034] running on GCE node
I20260928 23:52:34.620002 22385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:34.620623 22385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:34.621814 22385 hybrid_clock.cc:648] HybridClock initialized: now 1790639554621809 us; error 52 us; skew 500 ppm
Sep 28 23:52:34 dist-test-slave-p5bw krb5kdc[22144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639554, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:34.624462 22385 init.cc:377] Logged in from keytab as kudu/127.6.152.2@KRBTEST.COM (short username kudu)
I20260928 23:52:34.625432 22385 webserver.cc:460] Webserver started at http://127.6.152.2:44687/ using document root <none> and password file <none>
I20260928 23:52:34.626003 22385 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:34.626092 22385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:34.626327 22385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:52:34.628342 22385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "b5741f7f669e4e22b10ec6b0e1b486be"
format_stamp: "Formatted at 2026-09-28 23:52:34 on dist-test-slave-p5bw"
server_key: "550b514bff59f39c3aac255e6b82b4c2"
server_key_iv: "8123b9a7cacc6653ee97c0544f8cc9e2"
server_key_version: "encryptionkey@0"
I20260928 23:52:34.628973 22385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "b5741f7f669e4e22b10ec6b0e1b486be"
format_stamp: "Formatted at 2026-09-28 23:52:34 on dist-test-slave-p5bw"
server_key: "550b514bff59f39c3aac255e6b82b4c2"
server_key_iv: "8123b9a7cacc6653ee97c0544f8cc9e2"
server_key_version: "encryptionkey@0"
I20260928 23:52:34.632670 22385 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:52:34.635093 22400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:34.636263 22385 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:52:34.636405 22385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "b5741f7f669e4e22b10ec6b0e1b486be"
format_stamp: "Formatted at 2026-09-28 23:52:34 on dist-test-slave-p5bw"
server_key: "550b514bff59f39c3aac255e6b82b4c2"
server_key_iv: "8123b9a7cacc6653ee97c0544f8cc9e2"
server_key_version: "encryptionkey@0"
I20260928 23:52:34.636520 22385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:34.653079 22385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:34.655764 22385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:34.655963 22385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:34.656522 22385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:52:34.657377 22385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:52:34.657449 22385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:34.657516 22385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:52:34.657568 22385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:34.666488 22385 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:39253
I20260928 23:52:34.666512 22513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:39253 every 8 connection(s)
I20260928 23:52:34.667332 22385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:52:34.675711 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 22385
I20260928 23:52:34.675856 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:52:34.676141 6752 external_mini_cluster.cc:1428] Setting key 7f217b61d573d9b610860f7441a89ee8
Sep 28 23:52:34 dist-test-slave-p5bw krb5kdc[22144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639554, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:52:34.680836 22228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:34.669088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:33915 (local address 127.6.152.62:45073)
0928 23:52:34.669245 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:34.669249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:34.670043 (+ 794us) server_negotiation.cc:407] Connection header received
0928 23:52:34.671251 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:34.671253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:34.671299 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:34.671370 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:34.672403 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.672923 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.674008 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.674275 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.674367 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:34.676916 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:34.676924 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:34.676926 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:34.676949 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:34.678534 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:34.679131 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:34.679134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:34.679135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:34.679213 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:34.679628 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:34.679633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:34.679636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:34.679891 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:34.679997 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:34.680439 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 23:52:34.680570 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:52:34.681464 22516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:34.669479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.2:33915)
0928 23:52:34.669894 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:34.669935 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:34.671044 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:34.671493 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:34.671499 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:34.671829 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:34.672269 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:34.672282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.673019 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.673021 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:34.673864 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:34.673872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.674412 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.674414 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:34.674545 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:34.675038 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:34.675052 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:34.676723 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:34.678648 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:34.678657 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:34.678673 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:34.679033 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:34.679300 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:34.679306 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:34.679311 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:34.679513 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:34.679994 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:34.679998 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:34.680200 (+ 202us) client_negotiation.cc:769] Sending connection context
0928 23:52:34.680357 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 23:52:34.680579 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":120,"threads_started":1}
I20260928 23:52:34.682554 22514 heartbeater.cc:347] Connected to a master server at 127.6.152.62:45073
I20260928 23:52:34.682809 22514 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:34.683394 22514 heartbeater.cc:511] Master 127.6.152.62:45073 requested a full tablet report, sending...
I20260928 23:52:34.684525 22189 ts_manager.cc:194] Registered new tserver with Master: b5741f7f669e4e22b10ec6b0e1b486be (127.6.152.2:39253)
I20260928 23:52:34.685351 22189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.2@KRBTEST.COM'} at 127.6.152.2:33915
WARNING: no policy specified for kudu/127.6.152.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
I20260928 23:52:34.704977 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:45073
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:52:34.820009 22519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:34.820255 22519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:34.820323 22519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:34.824337 22519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:34.824393 22519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:34.824448 22519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:52:34.829717 22519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:45073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:34.830716 22519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:34.831693 22519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:34.840827 22527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:34.840875 22525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:34.840891 22524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:34.841008 22519 server_base.cc:1034] running on GCE node
I20260928 23:52:34.841462 22519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:34.842082 22519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:34.843400 22519 hybrid_clock.cc:648] HybridClock initialized: now 1790639554843380 us; error 36 us; skew 500 ppm
Sep 28 23:52:34 dist-test-slave-p5bw krb5kdc[22144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639554, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:34.845988 22519 init.cc:377] Logged in from keytab as kudu/127.6.152.3@KRBTEST.COM (short username kudu)
I20260928 23:52:34.846840 22519 webserver.cc:460] Webserver started at http://127.6.152.3:33547/ using document root <none> and password file <none>
I20260928 23:52:34.847426 22519 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:34.847514 22519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:34.847733 22519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:52:34.849840 22519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "51a1d42c90f7417e94cd5226098d0655"
format_stamp: "Formatted at 2026-09-28 23:52:34 on dist-test-slave-p5bw"
server_key: "134b9a4399ae32ca227e0b3d2389e1f7"
server_key_iv: "0893df6850936f37b4d4bccc89b3ace1"
server_key_version: "encryptionkey@0"
I20260928 23:52:34.850374 22519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "51a1d42c90f7417e94cd5226098d0655"
format_stamp: "Formatted at 2026-09-28 23:52:34 on dist-test-slave-p5bw"
server_key: "134b9a4399ae32ca227e0b3d2389e1f7"
server_key_iv: "0893df6850936f37b4d4bccc89b3ace1"
server_key_version: "encryptionkey@0"
I20260928 23:52:34.854017 22519 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:52:34.856200 22534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:34.857414 22519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:52:34.857548 22519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "51a1d42c90f7417e94cd5226098d0655"
format_stamp: "Formatted at 2026-09-28 23:52:34 on dist-test-slave-p5bw"
server_key: "134b9a4399ae32ca227e0b3d2389e1f7"
server_key_iv: "0893df6850936f37b4d4bccc89b3ace1"
server_key_version: "encryptionkey@0"
I20260928 23:52:34.857653 22519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:34.871125 22519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:34.874958 22519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:34.875161 22519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:34.875903 22519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:52:34.877118 22519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:52:34.877180 22519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:34.877240 22519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:52:34.877288 22519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:34.891194 22519 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:45739
I20260928 23:52:34.891250 22647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:45739 every 8 connection(s)
I20260928 23:52:34.892134 22519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:52:34.901189 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 22519
I20260928 23:52:34.901330 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:52:34.901602 6752 external_mini_cluster.cc:1428] Setting key 3961b069b38418e00854211709a3cbdd
Sep 28 23:52:34 dist-test-slave-p5bw krb5kdc[22144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639554, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:52:34.907421 22228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:34.894382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:56611 (local address 127.6.152.62:45073)
0928 23:52:34.894582 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:34.894589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:34.895618 (+ 1029us) server_negotiation.cc:407] Connection header received
0928 23:52:34.897117 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:34.897120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:34.897169 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:34.897254 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:34.898732 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.899506 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.900622 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.900917 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.901034 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:34.903874 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:34.903877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:34.903880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:34.903908 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:34.905439 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:34.905941 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:34.905945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:34.905947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:34.906028 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:34.906364 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:34.906369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:34.906370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:34.906548 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:34.906653 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:34.907130 (+ 477us) server_negotiation.cc:299] Negotiation successful
0928 23:52:34.907256 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:52:34.908221 22650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:34.894898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.3:56611)
0928 23:52:34.895461 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:34.895513 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:34.896878 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:34.897470 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:34.897477 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:34.897945 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:34.898583 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:34.898599 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.899623 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.899625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:34.900513 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:34.900521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.901074 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.901078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:34.901292 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:34.901954 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:34.901975 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:34.903717 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:34.905556 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:34.905562 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:34.905577 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:34.905836 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:34.906125 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:34.906128 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:34.906130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:34.906269 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:34.906650 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:34.906657 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:34.906903 (+ 246us) client_negotiation.cc:769] Sending connection context
0928 23:52:34.907091 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 23:52:34.907399 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":164,"threads_started":1}
I20260928 23:52:34.909384 22648 heartbeater.cc:347] Connected to a master server at 127.6.152.62:45073
I20260928 23:52:34.909665 22648 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:34.910192 22648 heartbeater.cc:511] Master 127.6.152.62:45073 requested a full tablet report, sending...
I20260928 23:52:34.911257 22189 ts_manager.cc:194] Registered new tserver with Master: 51a1d42c90f7417e94cd5226098d0655 (127.6.152.3:45739)
I20260928 23:52:34.911926 22189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.3@KRBTEST.COM'} at 127.6.152.3:56611
I20260928 23:52:34.915521 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:52:34.925414 22228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:34.917780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59596 (local address 127.6.152.62:45073)
0928 23:52:34.918258 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:34.918263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:34.918277 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:52:34.918320 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:34.918322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:34.918357 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:34.918421 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:34.919126 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.919762 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.920638 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.920928 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.921005 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:34.922329 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:34.922332 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:34.922334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:34.922355 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:34.923813 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:34.924207 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:34.924210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:34.924211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:34.924280 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:34.924505 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:34.924507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:34.924508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:34.924706 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:34.924804 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:34.925130 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 23:52:34.925285 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399}
I20260928 23:52:34.928099 22189 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:52:34.931301 22189 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:52:34.945436 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:34.940348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39940 (local address 127.6.152.2:39253)
0928 23:52:34.941261 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:34.941273 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:34.941294 (+ 21us) server_negotiation.cc:407] Connection header received
0928 23:52:34.941365 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:34.941370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:34.941536 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:34.941686 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:52:34.941925 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.942896 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.944186 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.944737 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.944898 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:34.945057 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:34.945186 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 23:52:34.945279 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":132,"threads_started":1}
I20260928 23:52:34.945425 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:34.940051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:39253 (local address 127.0.0.1:39940)
0928 23:52:34.940912 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:34.940925 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:34.940998 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:34.941690 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:34.941693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:34.941711 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:52:34.941803 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:34.941809 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.943021 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.943026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:34.944064 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:34.944073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.944902 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.944905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:34.945016 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:34.945052 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 23:52:34.945161 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 23:52:34.945284 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":95,"threads_started":1}
I20260928 23:52:34.945714 22661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:34.939705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:33381 (local address 127.0.0.1:47174)
0928 23:52:34.940586 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:34.940599 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:34.940697 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:34.941218 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:34.941221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:34.941239 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:52:34.941339 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:34.941346 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.942449 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.942453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:34.944027 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:34.944038 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.945184 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.945187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:34.945275 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:34.945304 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 23:52:34.945357 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 23:52:34.945425 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":70,"thread_start_us":101,"threads_started":1}
I20260928 23:52:34.945741 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:34.939363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:45739 (local address 127.0.0.1:43142)
0928 23:52:34.939697 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:34.939742 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:34.939860 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:34.940857 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:34.940863 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:34.940887 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:52:34.941040 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:34.941048 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.942664 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.942667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:34.944027 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:34.944038 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.945339 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.945345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:34.945422 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:34.945440 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 23:52:34.945502 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 23:52:34.945580 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":79,"thread_start_us":72,"threads_started":1}
I20260928 23:52:34.946153 22660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:34.939472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43142 (local address 127.6.152.3:45739)
0928 23:52:34.940249 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:34.940261 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:34.940288 (+ 27us) server_negotiation.cc:407] Connection header received
0928 23:52:34.940423 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:34.940430 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:34.940649 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:34.941193 (+ 544us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:52:34.941345 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.942535 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.944243 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.945031 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.945148 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:34.945281 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:34.945916 (+ 635us) server_negotiation.cc:299] Negotiation successful
0928 23:52:34.946006 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":175,"threads_started":1}
I20260928 23:52:34.946154 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:34.940135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47174 (local address 127.6.152.1:33381)
0928 23:52:34.940520 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:34.940534 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:34.940693 (+ 159us) server_negotiation.cc:407] Connection header received
0928 23:52:34.940832 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:34.940838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:34.941034 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:34.941247 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:52:34.941433 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.942320 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.944154 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.945031 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.945645 (+ 614us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:34.945909 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:34.945952 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 23:52:34.946028 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":119,"threads_started":1}
I20260928 23:52:34.948603 22448 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1a8e507b0c4a27a4eea03d5471048b (DEFAULT_TABLE table=test-table [id=3b95b39874814efc9f3068a45582fd56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:52:34.948606 22314 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1a8e507b0c4a27a4eea03d5471048b (DEFAULT_TABLE table=test-table [id=3b95b39874814efc9f3068a45582fd56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:52:34.949106 22569 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1a8e507b0c4a27a4eea03d5471048b (DEFAULT_TABLE table=test-table [id=3b95b39874814efc9f3068a45582fd56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:52:34.949743 22314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1a8e507b0c4a27a4eea03d5471048b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:52:34.949743 22448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1a8e507b0c4a27a4eea03d5471048b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:52:34.949997 22569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1a8e507b0c4a27a4eea03d5471048b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:52:34.956279 22665 tablet_bootstrap.cc:492] T 5e1a8e507b0c4a27a4eea03d5471048b P 51a1d42c90f7417e94cd5226098d0655: Bootstrap starting.
I20260928 23:52:34.957825 22666 tablet_bootstrap.cc:492] T 5e1a8e507b0c4a27a4eea03d5471048b P b5741f7f669e4e22b10ec6b0e1b486be: Bootstrap starting.
I20260928 23:52:34.957922 22667 tablet_bootstrap.cc:492] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753: Bootstrap starting.
I20260928 23:52:34.958904 22665 tablet_bootstrap.cc:654] T 5e1a8e507b0c4a27a4eea03d5471048b P 51a1d42c90f7417e94cd5226098d0655: Neither blocks nor log segments found. Creating new log.
I20260928 23:52:34.959942 22665 log.cc:824] T 5e1a8e507b0c4a27a4eea03d5471048b P 51a1d42c90f7417e94cd5226098d0655: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:34.960047 22667 tablet_bootstrap.cc:654] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753: Neither blocks nor log segments found. Creating new log.
I20260928 23:52:34.960472 22666 tablet_bootstrap.cc:654] T 5e1a8e507b0c4a27a4eea03d5471048b P b5741f7f669e4e22b10ec6b0e1b486be: Neither blocks nor log segments found. Creating new log.
I20260928 23:52:34.961362 22666 log.cc:824] T 5e1a8e507b0c4a27a4eea03d5471048b P b5741f7f669e4e22b10ec6b0e1b486be: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:34.961369 22667 log.cc:824] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:34.962066 22665 tablet_bootstrap.cc:492] T 5e1a8e507b0c4a27a4eea03d5471048b P 51a1d42c90f7417e94cd5226098d0655: No bootstrap required, opened a new log
I20260928 23:52:34.962265 22665 ts_tablet_manager.cc:1397] T 5e1a8e507b0c4a27a4eea03d5471048b P 51a1d42c90f7417e94cd5226098d0655: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:52:34.963459 22666 tablet_bootstrap.cc:492] T 5e1a8e507b0c4a27a4eea03d5471048b P b5741f7f669e4e22b10ec6b0e1b486be: No bootstrap required, opened a new log
I20260928 23:52:34.963657 22667 tablet_bootstrap.cc:492] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753: No bootstrap required, opened a new log
I20260928 23:52:34.963655 22666 ts_tablet_manager.cc:1397] T 5e1a8e507b0c4a27a4eea03d5471048b P b5741f7f669e4e22b10ec6b0e1b486be: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:52:34.963883 22667 ts_tablet_manager.cc:1397] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:52:34.965539 22665 raft_consensus.cc:348] T 5e1a8e507b0c4a27a4eea03d5471048b P 51a1d42c90f7417e94cd5226098d0655 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:52:34.965796 22665 raft_consensus.cc:374] T 5e1a8e507b0c4a27a4eea03d5471048b P 51a1d42c90f7417e94cd5226098d0655 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:52:34.965857 22665 raft_consensus.cc:729] T 5e1a8e507b0c4a27a4eea03d5471048b P 51a1d42c90f7417e94cd5226098d0655 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a1d42c90f7417e94cd5226098d0655, State: Initialized, Role: FOLLOWER
I20260928 23:52:34.966300 22665 consensus_queue.cc:260] T 5e1a8e507b0c4a27a4eea03d5471048b P 51a1d42c90f7417e94cd5226098d0655 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:52:34.966939 22648 heartbeater.cc:503] Master 127.6.152.62:45073 was elected leader, sending a full tablet report...
I20260928 23:52:34.967646 22667 raft_consensus.cc:348] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:52:34.967641 22666 raft_consensus.cc:348] T 5e1a8e507b0c4a27a4eea03d5471048b P b5741f7f669e4e22b10ec6b0e1b486be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:52:34.967904 22666 raft_consensus.cc:374] T 5e1a8e507b0c4a27a4eea03d5471048b P b5741f7f669e4e22b10ec6b0e1b486be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:52:34.967904 22667 raft_consensus.cc:374] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:52:34.967955 22666 raft_consensus.cc:729] T 5e1a8e507b0c4a27a4eea03d5471048b P b5741f7f669e4e22b10ec6b0e1b486be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5741f7f669e4e22b10ec6b0e1b486be, State: Initialized, Role: FOLLOWER
I20260928 23:52:34.967955 22667 raft_consensus.cc:729] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47fc0f3fa1e649e8ac43fb560b6fd753, State: Initialized, Role: FOLLOWER
I20260928 23:52:34.968302 22665 ts_tablet_manager.cc:1428] T 5e1a8e507b0c4a27a4eea03d5471048b P 51a1d42c90f7417e94cd5226098d0655: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260928 23:52:34.968438 22667 consensus_queue.cc:260] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:52:34.968438 22666 consensus_queue.cc:260] T 5e1a8e507b0c4a27a4eea03d5471048b P b5741f7f669e4e22b10ec6b0e1b486be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:52:34.969139 22380 heartbeater.cc:503] Master 127.6.152.62:45073 was elected leader, sending a full tablet report...
I20260928 23:52:34.969383 22667 ts_tablet_manager.cc:1428] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 23:52:34.969444 22514 heartbeater.cc:503] Master 127.6.152.62:45073 was elected leader, sending a full tablet report...
I20260928 23:52:34.970356 22666 ts_tablet_manager.cc:1428] T 5e1a8e507b0c4a27a4eea03d5471048b P b5741f7f669e4e22b10ec6b0e1b486be: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 23:52:34.972090 22672 raft_consensus.cc:482] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:52:34.972353 22672 raft_consensus.cc:504] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:52:34.973448 22672 leader_election.cc:288] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51a1d42c90f7417e94cd5226098d0655 (127.6.152.3:45739), b5741f7f669e4e22b10ec6b0e1b486be (127.6.152.2:39253)
I20260928 23:52:34.977475 22660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:34.973916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:39969 (local address 127.6.152.3:45739)
0928 23:52:34.974066 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:34.974071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:34.974238 (+ 167us) server_negotiation.cc:407] Connection header received
0928 23:52:34.974410 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:34.974413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:34.974461 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:34.974512 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:52:34.974909 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.975405 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.976437 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.976912 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.977028 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:34.977087 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:34.977288 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 23:52:34.977351 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 23:52:34.977522 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:34.973812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:45739 (local address 127.6.152.1:39969)
0928 23:52:34.974104 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:34.974118 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:34.974285 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:34.974665 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:34.974668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:34.974686 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:52:34.974820 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:34.974827 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.975514 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.975519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:34.976317 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:34.976326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.977029 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.977032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:34.977157 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:34.977190 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 23:52:34.977275 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 23:52:34.977344 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":20,"thread_start_us":131,"threads_started":1}
I20260928 23:52:34.977535 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:34.974296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:44927 (local address 127.6.152.2:39253)
0928 23:52:34.974419 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:34.974424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:34.974604 (+ 180us) server_negotiation.cc:407] Connection header received
0928 23:52:34.974670 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:34.974673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:34.974720 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:34.974790 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:52:34.975008 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.975492 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.976485 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.976974 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.977089 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:34.977154 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:34.977353 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 23:52:34.977417 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:52:34.977562 22675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:34.974222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:39253 (local address 127.6.152.1:44927)
0928 23:52:34.974501 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:34.974516 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:34.974605 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:34.974794 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:34.974796 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:34.974805 (+ 9us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:52:34.974900 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:34.974904 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.975605 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.975609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:34.976365 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:34.976372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.977092 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.977095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:34.977230 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:34.977247 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 23:52:34.977334 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 23:52:34.977403 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":3,"thread_start_us":101,"threads_started":1}
I20260928 23:52:34.978065 22468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1a8e507b0c4a27a4eea03d5471048b" candidate_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" is_pre_election: true
I20260928 23:52:34.978281 22584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1a8e507b0c4a27a4eea03d5471048b" candidate_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a1d42c90f7417e94cd5226098d0655" is_pre_election: true
I20260928 23:52:34.978333 22468 raft_consensus.cc:2454] T 5e1a8e507b0c4a27a4eea03d5471048b P b5741f7f669e4e22b10ec6b0e1b486be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47fc0f3fa1e649e8ac43fb560b6fd753 in term 0.
I20260928 23:52:34.978524 22584 raft_consensus.cc:2454] T 5e1a8e507b0c4a27a4eea03d5471048b P 51a1d42c90f7417e94cd5226098d0655 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47fc0f3fa1e649e8ac43fb560b6fd753 in term 0.
I20260928 23:52:34.978791 22267 leader_election.cc:302] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47fc0f3fa1e649e8ac43fb560b6fd753, b5741f7f669e4e22b10ec6b0e1b486be; no voters:
I20260928 23:52:34.979084 22672 raft_consensus.cc:2781] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:52:34.979189 22672 raft_consensus.cc:482] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:52:34.979247 22672 raft_consensus.cc:3037] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:52:34.980230 22672 raft_consensus.cc:504] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:52:34.980552 22672 leader_election.cc:288] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 1 election: Requested vote from peers 51a1d42c90f7417e94cd5226098d0655 (127.6.152.3:45739), b5741f7f669e4e22b10ec6b0e1b486be (127.6.152.2:39253)
I20260928 23:52:34.980881 22584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1a8e507b0c4a27a4eea03d5471048b" candidate_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a1d42c90f7417e94cd5226098d0655"
I20260928 23:52:34.981011 22584 raft_consensus.cc:3037] T 5e1a8e507b0c4a27a4eea03d5471048b P 51a1d42c90f7417e94cd5226098d0655 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:52:34.981014 22468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1a8e507b0c4a27a4eea03d5471048b" candidate_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5741f7f669e4e22b10ec6b0e1b486be"
I20260928 23:52:34.981143 22468 raft_consensus.cc:3037] T 5e1a8e507b0c4a27a4eea03d5471048b P b5741f7f669e4e22b10ec6b0e1b486be [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:52:34.982016 22584 raft_consensus.cc:2454] T 5e1a8e507b0c4a27a4eea03d5471048b P 51a1d42c90f7417e94cd5226098d0655 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47fc0f3fa1e649e8ac43fb560b6fd753 in term 1.
I20260928 23:52:34.982168 22468 raft_consensus.cc:2454] T 5e1a8e507b0c4a27a4eea03d5471048b P b5741f7f669e4e22b10ec6b0e1b486be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47fc0f3fa1e649e8ac43fb560b6fd753 in term 1.
I20260928 23:52:34.982322 22267 leader_election.cc:302] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47fc0f3fa1e649e8ac43fb560b6fd753, 51a1d42c90f7417e94cd5226098d0655; no voters:
I20260928 23:52:34.982486 22672 raft_consensus.cc:2781] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:52:34.982692 22672 raft_consensus.cc:686] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 LEADER]: Becoming Leader. State: Replica: 47fc0f3fa1e649e8ac43fb560b6fd753, State: Running, Role: LEADER
I20260928 23:52:34.983017 22672 consensus_queue.cc:237] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:52:34.986043 22188 catalog_manager.cc:5495] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47fc0f3fa1e649e8ac43fb560b6fd753 (127.6.152.1). New cstate: current_term: 1 leader_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } health_report { overall_health: HEALTHY } } }
I20260928 23:52:34.997574 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:34.993872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47186 (local address 127.6.152.1:33381)
0928 23:52:34.994002 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:34.994009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:34.994024 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:52:34.994211 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:34.994214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:34.994276 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:34.994428 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:52:34.994665 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.995161 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.996110 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:34.996341 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:34.996448 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:34.996753 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:52:34.997208 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:52:34.997349 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:34.997387 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 23:52:34.997438 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:52:35.003207 22468 raft_consensus.cc:1261] T 5e1a8e507b0c4a27a4eea03d5471048b P b5741f7f669e4e22b10ec6b0e1b486be [term 1 FOLLOWER]: Refusing update from remote peer 47fc0f3fa1e649e8ac43fb560b6fd753: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:52:35.003182 22584 raft_consensus.cc:1261] T 5e1a8e507b0c4a27a4eea03d5471048b P 51a1d42c90f7417e94cd5226098d0655 [term 1 FOLLOWER]: Refusing update from remote peer 47fc0f3fa1e649e8ac43fb560b6fd753: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:52:35.003923 22672 consensus_queue.cc:1035] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:52:35.004139 22676 consensus_queue.cc:1035] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:52:35.013671 22680 mvcc.cc:205] Tried to move back new op lower bound from 7334459617285824512 to 7334459617213308928. Current Snapshot: MvccSnapshot[applied={T|T < 7334459617285824512}]
I20260928 23:52:35.020702 22187 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:52:35.020903 22187 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:52:35.023403 22187 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43: Sending DeleteTablet for 3 replicas of tablet 5e1a8e507b0c4a27a4eea03d5471048b
I20260928 23:52:35.024255 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 22251
I20260928 23:52:35.024202 22314 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e1a8e507b0c4a27a4eea03d5471048b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:52:35 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:47174
I20260928 23:52:35.024405 22448 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e1a8e507b0c4a27a4eea03d5471048b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:52:35 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:39940
I20260928 23:52:35.024962 22569 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e1a8e507b0c4a27a4eea03d5471048b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:52:35 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:43142
I20260928 23:52:35.025436 22691 tablet_replica.cc:330] T 5e1a8e507b0c4a27a4eea03d5471048b P 51a1d42c90f7417e94cd5226098d0655: stopping tablet replica
I20260928 23:52:35.025664 22690 tablet_replica.cc:330] T 5e1a8e507b0c4a27a4eea03d5471048b P b5741f7f669e4e22b10ec6b0e1b486be: stopping tablet replica
I20260928 23:52:35.025872 22691 raft_consensus.cc:2229] T 5e1a8e507b0c4a27a4eea03d5471048b P 51a1d42c90f7417e94cd5226098d0655 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:52:35.025954 22690 raft_consensus.cc:2229] T 5e1a8e507b0c4a27a4eea03d5471048b P b5741f7f669e4e22b10ec6b0e1b486be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:52:35.026266 22691 raft_consensus.cc:2258] T 5e1a8e507b0c4a27a4eea03d5471048b P 51a1d42c90f7417e94cd5226098d0655 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:52:35.026265 22690 raft_consensus.cc:2258] T 5e1a8e507b0c4a27a4eea03d5471048b P b5741f7f669e4e22b10ec6b0e1b486be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:52:35.027680 22691 ts_tablet_manager.cc:1905] T 5e1a8e507b0c4a27a4eea03d5471048b P 51a1d42c90f7417e94cd5226098d0655: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:52:35.028050 22690 ts_tablet_manager.cc:1905] T 5e1a8e507b0c4a27a4eea03d5471048b P b5741f7f669e4e22b10ec6b0e1b486be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:52:35.029911 22691 ts_tablet_manager.cc:1918] T 5e1a8e507b0c4a27a4eea03d5471048b P 51a1d42c90f7417e94cd5226098d0655: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:52:35.030022 22691 log.cc:1196] T 5e1a8e507b0c4a27a4eea03d5471048b P 51a1d42c90f7417e94cd5226098d0655: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/5e1a8e507b0c4a27a4eea03d5471048b
I20260928 23:52:35.030308 22691 ts_tablet_manager.cc:1939] T 5e1a8e507b0c4a27a4eea03d5471048b P 51a1d42c90f7417e94cd5226098d0655: Deleting consensus metadata
I20260928 23:52:35.030635 22690 ts_tablet_manager.cc:1918] T 5e1a8e507b0c4a27a4eea03d5471048b P b5741f7f669e4e22b10ec6b0e1b486be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:52:35.030733 22690 log.cc:1196] T 5e1a8e507b0c4a27a4eea03d5471048b P b5741f7f669e4e22b10ec6b0e1b486be: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/wal/wals/5e1a8e507b0c4a27a4eea03d5471048b
I20260928 23:52:35.031032 22690 ts_tablet_manager.cc:1939] T 5e1a8e507b0c4a27a4eea03d5471048b P b5741f7f669e4e22b10ec6b0e1b486be: Deleting consensus metadata
I20260928 23:52:35.031555 22173 catalog_manager.cc:4841] TS 51a1d42c90f7417e94cd5226098d0655 (127.6.152.3:45739): tablet 5e1a8e507b0c4a27a4eea03d5471048b (table test-table [id=3b95b39874814efc9f3068a45582fd56]) successfully deleted
W20260928 23:52:35.032004 22174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.152.1:33381 (error 108)
I20260928 23:52:35.032114 22173 catalog_manager.cc:4841] TS b5741f7f669e4e22b10ec6b0e1b486be (127.6.152.2:39253): tablet 5e1a8e507b0c4a27a4eea03d5471048b (table test-table [id=3b95b39874814efc9f3068a45582fd56]) successfully deleted
I20260928 23:52:35.032256 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 22385
I20260928 23:52:35.033886 22661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:35.033541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:33381 (local address 127.0.0.1:47196)
0928 23:52:35.033706 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:35.033786 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.1:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 23:52:35.034065 22174 catalog_manager.cc:4584] TS 47fc0f3fa1e649e8ac43fb560b6fd753 (127.6.152.1:33381): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5e1a8e507b0c4a27a4eea03d5471048b: Network error: Client connection negotiation failed: client connection to 127.6.152.1:33381: connect: Connection refused (error 111)
I20260928 23:52:35.038105 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 22519
I20260928 23:52:35.043735 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 22157
I20260928 23:52:55.050366 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:45073
--webserver_interface=127.6.152.62
--webserver_port=40941
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:45073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:52:55.158900 22693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:55.159153 22693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:55.159194 22693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:55.163179 22693 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 23:52:55.163228 22693 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 23:52:55.163245 22693 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 23:52:55.163259 22693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:52:55.163308 22693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:55.163347 22693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:55.163375 22693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:52:55.163405 22693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:52:55.168577 22693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:45073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:45073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=40941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:55.169678 22693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:55.170686 22693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:55.183617 22699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:55.183631 22702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:55.183617 22700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:55.183663 22693 server_base.cc:1034] running on GCE node
I20260928 23:52:55.184558 22693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:55.185783 22693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:55.186959 22693 hybrid_clock.cc:648] HybridClock initialized: now 1790639575186944 us; error 27 us; skew 500 ppm
Sep 28 23:52:55 dist-test-slave-p5bw krb5kdc[22144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639575, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:55.194156 22693 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:52:55.195144 22693 webserver.cc:460] Webserver started at http://127.6.152.62:40941/ using document root <none> and password file <none>
I20260928 23:52:55.195740 22693 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:55.195849 22693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:55.199761 22693 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:52:55.201987 22709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:55.203225 22693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:52:55.203379 22693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "0ab3dc9959304dbb9b902fc81ace7f43"
format_stamp: "Formatted at 2026-09-28 23:52:34 on dist-test-slave-p5bw"
server_key: "466ede8102a6d02b8086f4f74b07eefc"
server_key_iv: "c1df7aba165e5e63b5d313373f6009d7"
server_key_version: "encryptionkey@0"
I20260928 23:52:55.203995 22693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:55.218211 22693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:55.221040 22693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:55.221242 22693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:55.228331 22693 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:45073
I20260928 23:52:55.228607 22761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:45073 every 8 connection(s)
I20260928 23:52:55.229492 22693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:52:55.234086 22762 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:52:55.235780 22762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43: Bootstrap starting.
I20260928 23:52:55.236646 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 22693
I20260928 23:52:55.237108 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:33381
--local_ip_for_outbound_sockets=127.6.152.1
--tserver_master_addrs=127.6.152.62:45073
--webserver_port=34805
--webserver_interface=127.6.152.1
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:52:55.253038 22762 log.cc:824] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:55.261479 22762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:52:55.261847 22762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43: Bootstrap complete.
I20260928 23:52:55.265875 22762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab3dc9959304dbb9b902fc81ace7f43" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45073 } }
I20260928 23:52:55.266269 22762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab3dc9959304dbb9b902fc81ace7f43, State: Initialized, Role: FOLLOWER
I20260928 23:52:55.266752 22762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab3dc9959304dbb9b902fc81ace7f43" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45073 } }
I20260928 23:52:55.266901 22762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:52:55.266992 22762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:52:55.267113 22762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:52:55.268955 22762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab3dc9959304dbb9b902fc81ace7f43" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45073 } }
I20260928 23:52:55.269387 22762 leader_election.cc:302] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ab3dc9959304dbb9b902fc81ace7f43; no voters:
I20260928 23:52:55.269706 22762 leader_election.cc:288] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 23:52:55.269781 22766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 23:52:55.270038 22766 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [term 2 LEADER]: Becoming Leader. State: Replica: 0ab3dc9959304dbb9b902fc81ace7f43, State: Running, Role: LEADER
I20260928 23:52:55.270372 22766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab3dc9959304dbb9b902fc81ace7f43" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45073 } }
I20260928 23:52:55.270911 22762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:52:55.271344 22766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ab3dc9959304dbb9b902fc81ace7f43. Latest consensus state: current_term: 2 leader_uuid: "0ab3dc9959304dbb9b902fc81ace7f43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab3dc9959304dbb9b902fc81ace7f43" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45073 } } }
I20260928 23:52:55.271451 22766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [sys.catalog]: This master's current role is: LEADER
I20260928 23:52:55.271837 22767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0ab3dc9959304dbb9b902fc81ace7f43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab3dc9959304dbb9b902fc81ace7f43" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45073 } } }
I20260928 23:52:55.271960 22767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [sys.catalog]: This master's current role is: LEADER
I20260928 23:52:55.271828 22770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:52:55.276580 22770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:52:55.277252 22770 catalog_manager.cc:1259] Loaded cluster ID: 5a5b082fa963486ea4bcf15a596c8950
I20260928 23:52:55.277329 22770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:52:55.278963 22770 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:52:55.279537 22770 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43: Loaded TSK: 0
I20260928 23:52:55.292451 22770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43: Generated new TSK 1
I20260928 23:52:55.292738 22770 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:52:55.355319 22764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:55.355567 22764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:55.355609 22764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:55.359659 22764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:55.359722 22764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:55.359812 22764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:52:55.364989 22764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:33381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=34805
--tserver_master_addrs=127.6.152.62:45073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:55.366007 22764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:55.366997 22764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:55.376652 22790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:55.376652 22789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:55.376688 22792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:55.377110 22764 server_base.cc:1034] running on GCE node
I20260928 23:52:55.377468 22764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:55.378149 22764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:55.379321 22764 hybrid_clock.cc:648] HybridClock initialized: now 1790639575379305 us; error 31 us; skew 500 ppm
Sep 28 23:52:55 dist-test-slave-p5bw krb5kdc[22144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639575, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:55.382186 22764 init.cc:377] Logged in from keytab as kudu/127.6.152.1@KRBTEST.COM (short username kudu)
I20260928 23:52:55.383239 22764 webserver.cc:460] Webserver started at http://127.6.152.1:34805/ using document root <none> and password file <none>
I20260928 23:52:55.383911 22764 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:55.384009 22764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:55.389041 22764 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:52:55.391633 22799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:55.393143 22764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:52:55.393266 22764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "47fc0f3fa1e649e8ac43fb560b6fd753"
format_stamp: "Formatted at 2026-09-28 23:52:34 on dist-test-slave-p5bw"
server_key: "7b1a85b8d117e3a44d10cccb9f0eb552"
server_key_iv: "1b7c7dc9a6b806fcd3ec615956a36197"
server_key_version: "encryptionkey@0"
I20260928 23:52:55.393985 22764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:55.421960 22764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:55.425565 22764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:55.425786 22764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:55.426497 22764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:52:55.427708 22806 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 23:52:55.431866 22764 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 23:52:55.431928 22764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.001s
I20260928 23:52:55.431998 22764 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 23:52:55.433779 22764 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 23:52:55.433836 22764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260928 23:52:55.434108 22806 tablet_bootstrap.cc:492] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753: Bootstrap starting.
I20260928 23:52:55.444639 22764 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:33381
I20260928 23:52:55.444644 22913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:33381 every 8 connection(s)
I20260928 23:52:55.445916 22764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:52:55.453837 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 22764
I20260928 23:52:55.454281 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:39253
--local_ip_for_outbound_sockets=127.6.152.2
--tserver_master_addrs=127.6.152.62:45073
--webserver_port=44687
--webserver_interface=127.6.152.2
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:52:55.454623 22806 log.cc:824] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:52:55 dist-test-slave-p5bw krb5kdc[22144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639575, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:52:55.462232 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:55.448135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.1:52503)
0928 23:52:55.448635 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:55.448676 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:55.449492 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:55.450558 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:55.450566 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:55.450973 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:55.451418 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:55.451431 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.452704 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.452707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:55.453689 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:55.453703 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.454708 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.454715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:55.454951 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:55.455655 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:55.455682 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:55.457136 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:55.459294 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:55.459298 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:55.459308 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:55.459546 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:55.459911 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:55.459919 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:55.459920 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:55.460042 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:55.460588 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:55.460594 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:55.460876 (+ 282us) client_negotiation.cc:769] Sending connection context
0928 23:52:55.461061 (+ 185us) client_negotiation.cc:240] Negotiation successful
0928 23:52:55.461373 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":135,"threads_started":1}
I20260928 23:52:55.462221 22916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:55.448096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:52503 (local address 127.6.152.62:45073)
0928 23:52:55.448465 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:55.448486 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:55.448792 (+ 306us) server_negotiation.cc:407] Connection header received
0928 23:52:55.449894 (+ 1102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:55.449905 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:55.450182 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:55.450505 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:55.451592 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.452545 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.453834 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.454537 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.454716 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:55.457334 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:55.457341 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:55.457358 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:55.457395 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:55.459126 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:55.459670 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:55.459676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:55.459681 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:55.459802 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:55.460144 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:55.460147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:55.460149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:55.460475 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:55.460615 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:55.461189 (+ 574us) server_negotiation.cc:299] Negotiation successful
0928 23:52:55.461458 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":115,"threads_started":1}
I20260928 23:52:55.464023 22806 tablet_bootstrap.cc:492] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:52:55.464476 22806 tablet_bootstrap.cc:492] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753: Bootstrap complete.
I20260928 23:52:55.464828 22914 heartbeater.cc:347] Connected to a master server at 127.6.152.62:45073
I20260928 23:52:55.465071 22914 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:55.465513 22914 heartbeater.cc:511] Master 127.6.152.62:45073 requested a full tablet report, sending...
I20260928 23:52:55.466032 22806 ts_tablet_manager.cc:1397] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.019s
I20260928 23:52:55.467710 22726 ts_manager.cc:194] Registered new tserver with Master: 47fc0f3fa1e649e8ac43fb560b6fd753 (127.6.152.1:33381)
I20260928 23:52:55.469005 22806 raft_consensus.cc:348] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:52:55.469640 22806 raft_consensus.cc:729] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47fc0f3fa1e649e8ac43fb560b6fd753, State: Initialized, Role: FOLLOWER
I20260928 23:52:55.470118 22806 consensus_queue.cc:260] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:52:55.471073 22806 ts_tablet_manager.cc:1428] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260928 23:52:55.471225 22726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.1@KRBTEST.COM'} at 127.6.152.1:52503
I20260928 23:52:55.473855 22914 heartbeater.cc:503] Master 127.6.152.62:45073 was elected leader, sending a full tablet report...
I20260928 23:52:55.477332 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:55.470825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:33381 (local address 127.0.0.1:47238)
0928 23:52:55.471877 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:55.471926 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:55.472053 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:55.473198 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:55.473204 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:55.473227 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:52:55.473339 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:55.473356 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.474740 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.474743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:55.475865 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:55.475876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.476950 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.476953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:55.477034 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:55.477061 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 23:52:55.477139 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 23:52:55.477227 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"thread_start_us":84,"threads_started":1}
I20260928 23:52:55.477393 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:55.471512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47238 (local address 127.6.152.1:33381)
0928 23:52:55.472298 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:55.472310 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:55.472330 (+ 20us) server_negotiation.cc:407] Connection header received
0928 23:52:55.472389 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:55.472394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:55.473040 (+ 646us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:55.473186 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:52:55.473464 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.474611 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.475978 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.476791 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.476948 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:55.477077 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:55.477177 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 23:52:55.477262 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"spinlock_wait_cycles":13056,"thread_start_us":102,"threads_started":1}
I20260928 23:52:55.478367 22848 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e1a8e507b0c4a27a4eea03d5471048b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:52:35 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:47238
I20260928 23:52:55.479126 22923 tablet_replica.cc:330] stopping tablet replica
I20260928 23:52:55.479410 22923 raft_consensus.cc:2229] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:52:55.479689 22923 raft_consensus.cc:2258] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:52:55.480562 22923 ts_tablet_manager.cc:1905] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:52:55.483937 22923 ts_tablet_manager.cc:1918] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:52:55.484014 22923 log.cc:1196] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/wal/wals/5e1a8e507b0c4a27a4eea03d5471048b
I20260928 23:52:55.484279 22923 ts_tablet_manager.cc:1939] T 5e1a8e507b0c4a27a4eea03d5471048b P 47fc0f3fa1e649e8ac43fb560b6fd753: Deleting consensus metadata
I20260928 23:52:55.485987 22711 catalog_manager.cc:4841] TS 47fc0f3fa1e649e8ac43fb560b6fd753 (127.6.152.1:33381): tablet 5e1a8e507b0c4a27a4eea03d5471048b (table test-table [id=3b95b39874814efc9f3068a45582fd56]) successfully deleted
W20260928 23:52:55.570779 22919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:55.571076 22919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:55.571175 22919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:55.575510 22919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:55.575587 22919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:55.575675 22919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:52:55.580799 22919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:39253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=44687
--tserver_master_addrs=127.6.152.62:45073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:55.581866 22919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:55.582841 22919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:55.592206 22931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:55.592259 22929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:55.592263 22928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:55.592396 22919 server_base.cc:1034] running on GCE node
I20260928 23:52:55.593016 22919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:55.593603 22919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:55.594781 22919 hybrid_clock.cc:648] HybridClock initialized: now 1790639575594764 us; error 27 us; skew 500 ppm
Sep 28 23:52:55 dist-test-slave-p5bw krb5kdc[22144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639575, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:55.597410 22919 init.cc:377] Logged in from keytab as kudu/127.6.152.2@KRBTEST.COM (short username kudu)
I20260928 23:52:55.598279 22919 webserver.cc:460] Webserver started at http://127.6.152.2:44687/ using document root <none> and password file <none>
I20260928 23:52:55.598855 22919 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:55.598964 22919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:55.602733 22919 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 23:52:55.604727 22938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:55.605919 22919 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 23:52:55.606060 22919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "b5741f7f669e4e22b10ec6b0e1b486be"
format_stamp: "Formatted at 2026-09-28 23:52:34 on dist-test-slave-p5bw"
server_key: "550b514bff59f39c3aac255e6b82b4c2"
server_key_iv: "8123b9a7cacc6653ee97c0544f8cc9e2"
server_key_version: "encryptionkey@0"
I20260928 23:52:55.606585 22919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:55.624745 22919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:55.627645 22919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:55.627866 22919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:55.628481 22919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:52:55.629330 22919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:52:55.629405 22919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:55.629478 22919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:52:55.629522 22919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:55.638615 22919 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:39253
I20260928 23:52:55.638650 23051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:39253 every 8 connection(s)
I20260928 23:52:55.639616 22919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:52:55.642058 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 22919
I20260928 23:52:55.642529 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:45739
--local_ip_for_outbound_sockets=127.6.152.3
--tserver_master_addrs=127.6.152.62:45073
--webserver_port=33547
--webserver_interface=127.6.152.3
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:52:55 dist-test-slave-p5bw krb5kdc[22144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639575, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:52:55.655119 22916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:55.641453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:42959 (local address 127.6.152.62:45073)
0928 23:52:55.641606 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:55.641610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:55.642568 (+ 958us) server_negotiation.cc:407] Connection header received
0928 23:52:55.644136 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:55.644139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:55.644201 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:55.644276 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:55.645787 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.646708 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.647933 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.648391 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.648519 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:55.651258 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:55.651264 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:55.651270 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:55.651308 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:55.653084 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:55.653584 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:55.653590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:55.653598 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:55.653700 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:55.654036 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:55.654042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:55.654045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:55.654280 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:55.654380 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:55.654818 (+ 438us) server_negotiation.cc:299] Negotiation successful
0928 23:52:55.654951 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 23:52:55.655915 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:55.641875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.2:42959)
0928 23:52:55.642407 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:55.642450 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:55.643879 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:55.644504 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:55.644515 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:55.645079 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:55.645621 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:55.645634 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.646851 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.646853 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:55.647786 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:55.647795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.648549 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.648552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:55.648694 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:55.649361 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:55.649377 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:55.651070 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:55.653218 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:55.653223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:55.653234 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:55.653482 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:55.653798 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:55.653800 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:55.653802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:55.653938 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:55.654377 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:55.654382 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:55.654602 (+ 220us) client_negotiation.cc:769] Sending connection context
0928 23:52:55.654753 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 23:52:55.655010 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":143,"threads_started":1}
I20260928 23:52:55.657173 23052 heartbeater.cc:347] Connected to a master server at 127.6.152.62:45073
I20260928 23:52:55.657429 23052 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:55.657964 23052 heartbeater.cc:511] Master 127.6.152.62:45073 requested a full tablet report, sending...
I20260928 23:52:55.659068 22726 ts_manager.cc:194] Registered new tserver with Master: b5741f7f669e4e22b10ec6b0e1b486be (127.6.152.2:39253)
I20260928 23:52:55.659644 22726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.2@KRBTEST.COM'} at 127.6.152.2:42959
W20260928 23:52:55.756485 23055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:52:55.756738 23055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:52:55.756779 23055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:52:55.760790 23055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:52:55.760924 23055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:52:55.761027 23055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:52:55.766176 23055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:45739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=33547
--tserver_master_addrs=127.6.152.62:45073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:52:55.767162 23055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:52:55.768174 23055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:52:55.777802 23060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:55.777871 23055 server_base.cc:1034] running on GCE node
W20260928 23:52:55.777810 23061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:52:55.777815 23063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:52:55.778424 23055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:52:55.779019 23055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:52:55.780203 23055 hybrid_clock.cc:648] HybridClock initialized: now 1790639575780169 us; error 49 us; skew 500 ppm
Sep 28 23:52:55 dist-test-slave-p5bw krb5kdc[22144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639575, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:52:55.782831 23055 init.cc:377] Logged in from keytab as kudu/127.6.152.3@KRBTEST.COM (short username kudu)
I20260928 23:52:55.783591 23055 webserver.cc:460] Webserver started at http://127.6.152.3:33547/ using document root <none> and password file <none>
I20260928 23:52:55.784138 23055 fs_manager.cc:359] Metadata directory not provided
I20260928 23:52:55.784206 23055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:52:55.787798 23055 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:52:55.789834 23070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:55.791079 23055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:52:55.791204 23055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "51a1d42c90f7417e94cd5226098d0655"
format_stamp: "Formatted at 2026-09-28 23:52:34 on dist-test-slave-p5bw"
server_key: "134b9a4399ae32ca227e0b3d2389e1f7"
server_key_iv: "0893df6850936f37b4d4bccc89b3ace1"
server_key_version: "encryptionkey@0"
I20260928 23:52:55.791718 23055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:52:55.804670 23055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:52:55.808137 23055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:52:55.808331 23055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:52:55.808964 23055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:52:55.809799 23055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:52:55.809854 23055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:55.809928 23055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:52:55.809962 23055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:52:55.818946 23055 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:45739
I20260928 23:52:55.818984 23183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:45739 every 8 connection(s)
I20260928 23:52:55.819991 23055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
Sep 28 23:52:55 dist-test-slave-p5bw krb5kdc[22144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639575, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:52:55.829432 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 23055
I20260928 23:52:55.833462 22916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:55.821856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:41127 (local address 127.6.152.62:45073)
0928 23:52:55.821992 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:55.821995 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:55.822809 (+ 814us) server_negotiation.cc:407] Connection header received
0928 23:52:55.824099 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:55.824102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:55.824156 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:55.824235 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:55.825429 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.825951 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.827068 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.827296 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.827416 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:55.829779 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:55.829783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:55.829786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:55.829812 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:55.831327 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:55.831891 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:55.831894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:55.831895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:55.831970 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:55.832346 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:55.832350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:55.832352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:55.832572 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:55.832670 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:55.833184 (+ 514us) server_negotiation.cc:299] Negotiation successful
0928 23:52:55.833307 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 23:52:55.834256 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:55.822256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.3:41127)
0928 23:52:55.822668 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:55.822708 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:55.823893 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:55.824404 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:55.824410 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:55.824744 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:52:55.825283 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:55.825296 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.826075 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.826080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:55.826944 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:55.826953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.827449 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.827453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:55.827626 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:55.828135 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:52:55.828150 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:52:55.829627 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:52:55.831451 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:55.831458 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:55.831474 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:55.831791 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:55.832062 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:52:55.832065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:52:55.832067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:52:55.832223 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:52:55.832674 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:52:55.832678 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:52:55.832952 (+ 274us) client_negotiation.cc:769] Sending connection context
0928 23:52:55.833104 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 23:52:55.833373 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":110,"threads_started":1}
I20260928 23:52:55.835557 23184 heartbeater.cc:347] Connected to a master server at 127.6.152.62:45073
I20260928 23:52:55.835808 23184 heartbeater.cc:464] Registering TS with master...
I20260928 23:52:55.836272 23184 heartbeater.cc:511] Master 127.6.152.62:45073 requested a full tablet report, sending...
I20260928 23:52:55.837347 22726 ts_manager.cc:194] Registered new tserver with Master: 51a1d42c90f7417e94cd5226098d0655 (127.6.152.3:45739)
I20260928 23:52:55.838033 22726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.3@KRBTEST.COM'} at 127.6.152.3:41127
I20260928 23:52:55.838773 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:55.830090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36200 (local address 127.6.152.62:45073)
0928 23:52:55.830447 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:55.830453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:55.830470 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:52:55.830568 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:55.830573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:55.830626 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:55.830709 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:55.831487 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.832266 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.833383 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.833692 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.833792 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:55.835369 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:55.835372 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:55.835374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:55.835393 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:55.836903 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:55.837381 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:55.837383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:55.837384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:55.837485 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:55.837741 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:55.837743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:55.837744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:55.837898 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:55.838044 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:55.838549 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 23:52:55.838661 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":91,"threads_started":1}
I20260928 23:52:55.840390 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:52:55.843977 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:55.841047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36202 (local address 127.6.152.62:45073)
0928 23:52:55.841187 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:55.841191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:55.841327 (+ 136us) server_negotiation.cc:407] Connection header received
0928 23:52:55.841455 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:55.841457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:55.841541 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:55.841615 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:52:55.841773 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.842226 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.843017 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.843247 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.843326 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:55.843570 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:52:55.843684 (+ 114us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 23:52:55.843859 (+ 175us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36202: token expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 23:52:55.844039 23188 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36202: token expired
W20260928 23:52:55.844408 22651 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.152.62:45073)
I20260928 23:52:55.850862 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:55.844732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36214 (local address 127.6.152.62:45073)
0928 23:52:55.844865 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:55.844871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:55.844884 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:52:55.844960 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:55.844962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:55.844991 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:55.845049 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:52:55.845569 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.846009 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.846806 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.846997 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.847071 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:55.848266 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:52:55.848268 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:52:55.848270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:52:55.848287 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:52:55.849599 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:55.849952 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:55.849954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:55.849955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:55.850024 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:52:55.850235 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:52:55.850237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:52:55.850238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:52:55.850412 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:52:55.850510 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:55.850687 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 23:52:55.850769 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:52:55.853003 22726 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:52:55.856053 22726 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:52:55.863375 22848 tablet_service.cc:1467] Processing CreateTablet for tablet 1633631cb6974f36b3fe5d8be4b8201b (DEFAULT_TABLE table=test-table [id=87bed605a25f46c8bd960e8e2bc04493]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:52:55.864001 22848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1633631cb6974f36b3fe5d8be4b8201b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:52:55.866592 22806 tablet_bootstrap.cc:492] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753: Bootstrap starting.
I20260928 23:52:55.867532 22806 tablet_bootstrap.cc:654] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753: Neither blocks nor log segments found. Creating new log.
I20260928 23:52:55.868274 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:55.862325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:45739 (local address 127.0.0.1:35800)
0928 23:52:55.862661 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:55.862676 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:55.862748 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:55.863731 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:55.863734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:55.863754 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:52:55.863997 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:55.864003 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.865330 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.865333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:55.866643 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:55.866648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.867763 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.867769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:55.867870 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:55.867902 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 23:52:55.868009 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 23:52:55.868109 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":265}
I20260928 23:52:55.868289 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:55.862994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35800 (local address 127.6.152.3:45739)
0928 23:52:55.863360 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:55.863370 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:55.863389 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:52:55.863448 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:55.863452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:55.863590 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:55.863813 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:52:55.864119 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.865194 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.866775 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.867598 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.867751 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:55.867897 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:55.868047 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 23:52:55.868146 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":158,"threads_started":1}
I20260928 23:52:55.868458 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:55.862521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:39253 (local address 127.0.0.1:32892)
0928 23:52:55.862934 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:55.862947 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:55.863022 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:55.863677 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:55.863680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:55.863700 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:52:55.863824 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:55.863830 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.865294 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.865300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:55.866635 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:55.866647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.868090 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.868093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:55.868180 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:55.868202 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:52:55.868250 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 23:52:55.868328 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":69,"thread_start_us":133,"threads_started":1}
I20260928 23:52:55.869031 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:55.862722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32892 (local address 127.6.152.2:39253)
0928 23:52:55.863165 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:55.863200 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:55.863228 (+ 28us) server_negotiation.cc:407] Connection header received
0928 23:52:55.863308 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:55.863313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:55.863542 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:55.863707 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:52:55.863925 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.865183 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.866775 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:55.867956 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:55.868597 (+ 641us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:55.868737 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:55.868782 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 23:52:55.868902 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":105,"threads_started":1}
I20260928 23:52:55.869408 22806 tablet_bootstrap.cc:492] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753: No bootstrap required, opened a new log
I20260928 23:52:55.869513 22806 ts_tablet_manager.cc:1397] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 23:52:55.869958 22806 raft_consensus.cc:348] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:52:55.870077 22806 raft_consensus.cc:374] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:52:55.870114 22806 raft_consensus.cc:729] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47fc0f3fa1e649e8ac43fb560b6fd753, State: Initialized, Role: FOLLOWER
I20260928 23:52:55.870580 22806 consensus_queue.cc:260] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:52:55.871328 23118 tablet_service.cc:1467] Processing CreateTablet for tablet 1633631cb6974f36b3fe5d8be4b8201b (DEFAULT_TABLE table=test-table [id=87bed605a25f46c8bd960e8e2bc04493]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:52:55.871883 22806 ts_tablet_manager.cc:1428] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 23:52:55.871963 22986 tablet_service.cc:1467] Processing CreateTablet for tablet 1633631cb6974f36b3fe5d8be4b8201b (DEFAULT_TABLE table=test-table [id=87bed605a25f46c8bd960e8e2bc04493]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:52:55.872442 23118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1633631cb6974f36b3fe5d8be4b8201b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:52:55.873095 22986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1633631cb6974f36b3fe5d8be4b8201b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:52:55.881765 23197 tablet_bootstrap.cc:492] T 1633631cb6974f36b3fe5d8be4b8201b P 51a1d42c90f7417e94cd5226098d0655: Bootstrap starting.
I20260928 23:52:55.882477 23198 tablet_bootstrap.cc:492] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be: Bootstrap starting.
I20260928 23:52:55.884155 23197 tablet_bootstrap.cc:654] T 1633631cb6974f36b3fe5d8be4b8201b P 51a1d42c90f7417e94cd5226098d0655: Neither blocks nor log segments found. Creating new log.
I20260928 23:52:55.884577 23198 tablet_bootstrap.cc:654] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be: Neither blocks nor log segments found. Creating new log.
I20260928 23:52:55.885018 23197 log.cc:824] T 1633631cb6974f36b3fe5d8be4b8201b P 51a1d42c90f7417e94cd5226098d0655: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:55.885334 23198 log.cc:824] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be: Log is configured to *not* fsync() on all Append() calls
I20260928 23:52:55.886999 23197 tablet_bootstrap.cc:492] T 1633631cb6974f36b3fe5d8be4b8201b P 51a1d42c90f7417e94cd5226098d0655: No bootstrap required, opened a new log
I20260928 23:52:55.887107 23198 tablet_bootstrap.cc:492] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be: No bootstrap required, opened a new log
I20260928 23:52:55.887197 23197 ts_tablet_manager.cc:1397] T 1633631cb6974f36b3fe5d8be4b8201b P 51a1d42c90f7417e94cd5226098d0655: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:52:55.887250 23198 ts_tablet_manager.cc:1397] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 23:52:55.891075 23197 raft_consensus.cc:348] T 1633631cb6974f36b3fe5d8be4b8201b P 51a1d42c90f7417e94cd5226098d0655 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:52:55.891076 23198 raft_consensus.cc:348] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:52:55.891367 23197 raft_consensus.cc:374] T 1633631cb6974f36b3fe5d8be4b8201b P 51a1d42c90f7417e94cd5226098d0655 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:52:55.891360 23198 raft_consensus.cc:374] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:52:55.891422 23197 raft_consensus.cc:729] T 1633631cb6974f36b3fe5d8be4b8201b P 51a1d42c90f7417e94cd5226098d0655 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a1d42c90f7417e94cd5226098d0655, State: Initialized, Role: FOLLOWER
I20260928 23:52:55.891422 23198 raft_consensus.cc:729] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5741f7f669e4e22b10ec6b0e1b486be, State: Initialized, Role: FOLLOWER
I20260928 23:52:55.891891 23198 consensus_queue.cc:260] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:52:55.891891 23197 consensus_queue.cc:260] T 1633631cb6974f36b3fe5d8be4b8201b P 51a1d42c90f7417e94cd5226098d0655 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:52:55.892534 23052 heartbeater.cc:503] Master 127.6.152.62:45073 was elected leader, sending a full tablet report...
I20260928 23:52:55.892771 23184 heartbeater.cc:503] Master 127.6.152.62:45073 was elected leader, sending a full tablet report...
I20260928 23:52:55.892915 23198 ts_tablet_manager.cc:1428] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260928 23:52:55.893056 23197 ts_tablet_manager.cc:1428] T 1633631cb6974f36b3fe5d8be4b8201b P 51a1d42c90f7417e94cd5226098d0655: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260928 23:52:55.947256 22915 tablet.cc:2369] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:52:56.070964 23185 tablet.cc:2369] T 1633631cb6974f36b3fe5d8be4b8201b P 51a1d42c90f7417e94cd5226098d0655: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:52:56.140784 23053 tablet.cc:2369] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:52:56.276132 23202 raft_consensus.cc:482] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:52:56.276340 23202 raft_consensus.cc:504] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:52:56.277505 23202 leader_election.cc:288] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51a1d42c90f7417e94cd5226098d0655 (127.6.152.3:45739), 47fc0f3fa1e649e8ac43fb560b6fd753 (127.6.152.1:33381)
I20260928 23:52:56.281150 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:56.277857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:46443 (local address 127.6.152.3:45739)
0928 23:52:56.278021 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:56.278027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:56.278060 (+ 33us) server_negotiation.cc:407] Connection header received
0928 23:52:56.278229 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:56.278234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:56.278283 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:56.278387 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:52:56.278656 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:56.279148 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:56.280196 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:56.280593 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:56.280710 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:56.280772 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:56.280969 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 23:52:56.281029 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 23:52:56.281169 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:56.277716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:45739 (local address 127.6.152.2:46443)
0928 23:52:56.277979 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:56.277995 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:56.278097 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:56.278402 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:56.278405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:56.278423 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:52:56.278544 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:56.278550 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:56.279261 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:56.279265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:56.280069 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:56.280078 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:56.280712 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:56.280716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:56.280848 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:56.280866 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 23:52:56.280954 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 23:52:56.281024 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":14,"thread_start_us":74,"threads_started":1}
I20260928 23:52:56.281503 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:56.277838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:33381 (local address 127.6.152.2:37263)
0928 23:52:56.278232 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:56.278247 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:52:56.278375 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:52:56.278658 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:52:56.278660 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:52:56.278676 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:52:56.278807 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:56.278813 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:56.279525 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:56.279530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:56.280315 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:52:56.280324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:52:56.281054 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:52:56.281057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:52:56.281158 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:56.281190 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 23:52:56.281286 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 23:52:56.281371 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":94,"threads_started":1}
I20260928 23:52:56.281605 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:56.278034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:37263 (local address 127.6.152.1:33381)
0928 23:52:56.278261 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:56.278267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:56.278347 (+ 80us) server_negotiation.cc:407] Connection header received
0928 23:52:56.278507 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:56.278509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:56.278554 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:56.278648 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:52:56.278924 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:56.279413 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:56.280444 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:56.280939 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:56.281054 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:56.281114 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:56.281388 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 23:52:56.281469 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":64,"threads_started":1}
I20260928 23:52:56.281812 23138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1633631cb6974f36b3fe5d8be4b8201b" candidate_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a1d42c90f7417e94cd5226098d0655" is_pre_election: true
I20260928 23:52:56.282136 23138 raft_consensus.cc:2454] T 1633631cb6974f36b3fe5d8be4b8201b P 51a1d42c90f7417e94cd5226098d0655 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5741f7f669e4e22b10ec6b0e1b486be in term 0.
I20260928 23:52:56.282127 22868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1633631cb6974f36b3fe5d8be4b8201b" candidate_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" is_pre_election: true
I20260928 23:52:56.282403 22868 raft_consensus.cc:2454] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5741f7f669e4e22b10ec6b0e1b486be in term 0.
I20260928 23:52:56.282624 22939 leader_election.cc:302] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51a1d42c90f7417e94cd5226098d0655, b5741f7f669e4e22b10ec6b0e1b486be; no voters:
I20260928 23:52:56.282856 23202 raft_consensus.cc:2781] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:52:56.282938 23202 raft_consensus.cc:482] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:52:56.282979 23202 raft_consensus.cc:3037] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:52:56.283998 23202 raft_consensus.cc:504] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:52:56.284334 23202 leader_election.cc:288] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be [CANDIDATE]: Term 1 election: Requested vote from peers 51a1d42c90f7417e94cd5226098d0655 (127.6.152.3:45739), 47fc0f3fa1e649e8ac43fb560b6fd753 (127.6.152.1:33381)
I20260928 23:52:56.284754 23138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1633631cb6974f36b3fe5d8be4b8201b" candidate_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a1d42c90f7417e94cd5226098d0655"
I20260928 23:52:56.284906 23138 raft_consensus.cc:3037] T 1633631cb6974f36b3fe5d8be4b8201b P 51a1d42c90f7417e94cd5226098d0655 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:52:56.284909 22868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1633631cb6974f36b3fe5d8be4b8201b" candidate_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753"
I20260928 23:52:56.285054 22868 raft_consensus.cc:3037] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:52:56.285951 23138 raft_consensus.cc:2454] T 1633631cb6974f36b3fe5d8be4b8201b P 51a1d42c90f7417e94cd5226098d0655 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5741f7f669e4e22b10ec6b0e1b486be in term 1.
I20260928 23:52:56.286096 22868 raft_consensus.cc:2454] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5741f7f669e4e22b10ec6b0e1b486be in term 1.
I20260928 23:52:56.286250 22939 leader_election.cc:302] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51a1d42c90f7417e94cd5226098d0655, b5741f7f669e4e22b10ec6b0e1b486be; no voters:
I20260928 23:52:56.286412 23202 raft_consensus.cc:2781] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:52:56.286648 23202 raft_consensus.cc:686] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be [term 1 LEADER]: Becoming Leader. State: Replica: b5741f7f669e4e22b10ec6b0e1b486be, State: Running, Role: LEADER
I20260928 23:52:56.286981 23202 consensus_queue.cc:237] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:52:56.290251 22724 catalog_manager.cc:5495] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be reported cstate change: term changed from 0 to 1, leader changed from <none> to b5741f7f669e4e22b10ec6b0e1b486be (127.6.152.2). New cstate: current_term: 1 leader_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } health_report { overall_health: UNKNOWN } } }
I20260928 23:52:56.321802 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:56.318298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32908 (local address 127.6.152.2:39253)
0928 23:52:56.318452 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 23:52:56.318457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:52:56.318472 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:52:56.318613 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:52:56.318615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:52:56.318671 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:52:56.318784 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:52:56.319019 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:56.319520 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:56.320362 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:52:56.320576 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:52:56.320686 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:52:56.321017 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:52:56.321430 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:52:56.321566 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 23:52:56.321607 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 23:52:56.321661 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 23:52:56.327442 23138 raft_consensus.cc:1261] T 1633631cb6974f36b3fe5d8be4b8201b P 51a1d42c90f7417e94cd5226098d0655 [term 1 FOLLOWER]: Refusing update from remote peer b5741f7f669e4e22b10ec6b0e1b486be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:52:56.327437 22868 raft_consensus.cc:1261] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Refusing update from remote peer b5741f7f669e4e22b10ec6b0e1b486be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:52:56.328178 23206 consensus_queue.cc:1035] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be [LEADER]: Connected to new peer: Peer: permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:52:56.328411 23202 consensus_queue.cc:1035] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be [LEADER]: Connected to new peer: Peer: permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:52:56.334885 23209 mvcc.cc:205] Tried to move back new op lower bound from 7334459704630398976 to 7334459704474054656. Current Snapshot: MvccSnapshot[applied={T|T < 7334459704630398976}]
I20260928 23:52:56.335153 23210 mvcc.cc:205] Tried to move back new op lower bound from 7334459704630398976 to 7334459704474054656. Current Snapshot: MvccSnapshot[applied={T|T < 7334459704630398976}]
I20260928 23:52:56.343405 22724 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:52:56.343546 22724 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:52:56.345716 22724 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43: Sending DeleteTablet for 3 replicas of tablet 1633631cb6974f36b3fe5d8be4b8201b
I20260928 23:52:56.346356 23118 tablet_service.cc:1514] Processing DeleteTablet for tablet 1633631cb6974f36b3fe5d8be4b8201b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:52:56 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:35800
I20260928 23:52:56.346532 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 22764
I20260928 23:52:56.346664 22986 tablet_service.cc:1514] Processing DeleteTablet for tablet 1633631cb6974f36b3fe5d8be4b8201b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:52:56 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:32892
I20260928 23:52:56.348096 23219 tablet_replica.cc:330] T 1633631cb6974f36b3fe5d8be4b8201b P 51a1d42c90f7417e94cd5226098d0655: stopping tablet replica
I20260928 23:52:56.348392 23219 raft_consensus.cc:2229] T 1633631cb6974f36b3fe5d8be4b8201b P 51a1d42c90f7417e94cd5226098d0655 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:52:56.348690 23219 raft_consensus.cc:2258] T 1633631cb6974f36b3fe5d8be4b8201b P 51a1d42c90f7417e94cd5226098d0655 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:52:56.349015 23220 tablet_replica.cc:330] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be: stopping tablet replica
I20260928 23:52:56.349354 23220 raft_consensus.cc:2229] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:52:56.349740 23220 raft_consensus.cc:2258] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:52:56.350466 23219 ts_tablet_manager.cc:1905] T 1633631cb6974f36b3fe5d8be4b8201b P 51a1d42c90f7417e94cd5226098d0655: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:52:56.351300 23220 ts_tablet_manager.cc:1905] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 23:52:56.353386 22711 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.152.1:33381 (error 108)
I20260928 23:52:56.353412 23219 ts_tablet_manager.cc:1918] T 1633631cb6974f36b3fe5d8be4b8201b P 51a1d42c90f7417e94cd5226098d0655: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:52:56.353499 23219 log.cc:1196] T 1633631cb6974f36b3fe5d8be4b8201b P 51a1d42c90f7417e94cd5226098d0655: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/1633631cb6974f36b3fe5d8be4b8201b
I20260928 23:52:56.353812 23219 ts_tablet_manager.cc:1939] T 1633631cb6974f36b3fe5d8be4b8201b P 51a1d42c90f7417e94cd5226098d0655: Deleting consensus metadata
I20260928 23:52:56.354103 23220 ts_tablet_manager.cc:1918] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:52:56.354218 23220 log.cc:1196] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/wal/wals/1633631cb6974f36b3fe5d8be4b8201b
I20260928 23:52:56.354565 23220 ts_tablet_manager.cc:1939] T 1633631cb6974f36b3fe5d8be4b8201b P b5741f7f669e4e22b10ec6b0e1b486be: Deleting consensus metadata
I20260928 23:52:56.354622 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 22919
I20260928 23:52:56.355911 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:56.355632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:33381 (local address 127.0.0.1:47244)
0928 23:52:56.355768 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:56.355841 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.1:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 23:52:56.356021 22710 catalog_manager.cc:4841] TS 51a1d42c90f7417e94cd5226098d0655 (127.6.152.3:45739): tablet 1633631cb6974f36b3fe5d8be4b8201b (table test-table [id=87bed605a25f46c8bd960e8e2bc04493]) successfully deleted
W20260928 23:52:56.356184 22711 catalog_manager.cc:4584] TS 47fc0f3fa1e649e8ac43fb560b6fd753 (127.6.152.1:33381): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1633631cb6974f36b3fe5d8be4b8201b: Network error: Client connection negotiation failed: client connection to 127.6.152.1:33381: connect: Connection refused (error 111)
I20260928 23:52:56.363011 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 23055
I20260928 23:52:56.363250 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:52:56.362929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:39253 (local address 127.0.0.1:32912)
0928 23:52:56.363072 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 23:52:56.363150 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.2:39253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 23:52:56.368667 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 22693
I20260928 23:53:16.375345 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:45073
--webserver_interface=127.6.152.62
--webserver_port=40941
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:45073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:53:16.483209 23222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:16.483453 23222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:16.483492 23222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:16.487440 23222 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 23:53:16.487490 23222 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 23:53:16.487505 23222 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 23:53:16.487519 23222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:53:16.487537 23222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:16.487555 23222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:16.487612 23222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:53:16.487634 23222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:53:16.492698 23222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:45073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:45073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=40941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:16.493777 23222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:16.494777 23222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:16.503580 23228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:16.503705 23230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:16.503705 23227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:16.503755 23222 server_base.cc:1034] running on GCE node
I20260928 23:53:16.504498 23222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:16.505640 23222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:16.506801 23222 hybrid_clock.cc:648] HybridClock initialized: now 1790639596506769 us; error 48 us; skew 500 ppm
Sep 28 23:53:16 dist-test-slave-p5bw krb5kdc[22144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639596, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:16.515877 23222 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:53:16.516958 23222 webserver.cc:460] Webserver started at http://127.6.152.62:40941/ using document root <none> and password file <none>
I20260928 23:53:16.517583 23222 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:16.517647 23222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:16.521530 23222 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 23:53:16.523831 23237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:16.525136 23222 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:53:16.525260 23222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "0ab3dc9959304dbb9b902fc81ace7f43"
format_stamp: "Formatted at 2026-09-28 23:52:34 on dist-test-slave-p5bw"
server_key: "466ede8102a6d02b8086f4f74b07eefc"
server_key_iv: "c1df7aba165e5e63b5d313373f6009d7"
server_key_version: "encryptionkey@0"
I20260928 23:53:16.525841 23222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:16.540678 23222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:16.543879 23222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:16.544094 23222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:16.551651 23222 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:45073
I20260928 23:53:16.551959 23289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:45073 every 8 connection(s)
I20260928 23:53:16.552785 23222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:53:16.557072 23290 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:53:16.558763 23290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43: Bootstrap starting.
I20260928 23:53:16.561827 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 23222
I20260928 23:53:16.562294 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:33381
--local_ip_for_outbound_sockets=127.6.152.1
--tserver_master_addrs=127.6.152.62:45073
--webserver_port=34805
--webserver_interface=127.6.152.1
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:53:16.576982 23290 log.cc:824] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:16.587975 23290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:53:16.588375 23290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43: Bootstrap complete.
I20260928 23:53:16.592357 23290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab3dc9959304dbb9b902fc81ace7f43" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45073 } }
I20260928 23:53:16.592726 23290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab3dc9959304dbb9b902fc81ace7f43, State: Initialized, Role: FOLLOWER
I20260928 23:53:16.593221 23290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab3dc9959304dbb9b902fc81ace7f43" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45073 } }
I20260928 23:53:16.593343 23290 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:53:16.593432 23290 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:53:16.593523 23290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [term 2 FOLLOWER]: Advancing to term 3
I20260928 23:53:16.595358 23290 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab3dc9959304dbb9b902fc81ace7f43" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45073 } }
I20260928 23:53:16.595680 23290 leader_election.cc:302] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ab3dc9959304dbb9b902fc81ace7f43; no voters:
I20260928 23:53:16.596009 23290 leader_election.cc:288] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 23:53:16.596091 23294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 23:53:16.596280 23294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [term 3 LEADER]: Becoming Leader. State: Replica: 0ab3dc9959304dbb9b902fc81ace7f43, State: Running, Role: LEADER
I20260928 23:53:16.596699 23294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab3dc9959304dbb9b902fc81ace7f43" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45073 } }
I20260928 23:53:16.597278 23290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:53:16.597666 23294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ab3dc9959304dbb9b902fc81ace7f43. Latest consensus state: current_term: 3 leader_uuid: "0ab3dc9959304dbb9b902fc81ace7f43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab3dc9959304dbb9b902fc81ace7f43" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45073 } } }
I20260928 23:53:16.597869 23294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:16.598042 23295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0ab3dc9959304dbb9b902fc81ace7f43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab3dc9959304dbb9b902fc81ace7f43" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45073 } } }
I20260928 23:53:16.598160 23295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43 [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:16.598412 23298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:53:16.603590 23298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:53:16.604311 23298 catalog_manager.cc:1259] Loaded cluster ID: 5a5b082fa963486ea4bcf15a596c8950
I20260928 23:53:16.604393 23298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:53:16.606199 23298 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:53:16.606897 23298 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43: Loaded TSK: 0
I20260928 23:53:16.606980 23298 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0ab3dc9959304dbb9b902fc81ace7f43: Loaded TSK: 1
F20260928 23:53:16.607561 23298 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0e8d25f9ad google::LogMessage::Fail() at ??:0
@ 0x7f0e8d26288a google::LogMessage::SendToLog() at ??:0
@ 0x7f0e8d25f4de google::LogMessage::Flush() at ??:0
@ 0x7f0e8d260a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0e8ff2318d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f0e8ff1bf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f0e97bc47bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f0e97b9e99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f0e97b9f178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f0e97be0df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f0e97bff82b std::function<>::operator()() at ??:0
@ 0x7f0e97b9eb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f0e97b9ff3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0e97b9c579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f0e97be0640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f0e8f4b13d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0e8f4b1c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f0e8f4b33f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f0e8f4a2512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0e90ae66db start_thread at ??:0
@ 0x7f0e8bd5d61f clone at ??:0
W20260928 23:53:16.694612 23292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:16.694852 23292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:16.694896 23292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:16.699221 23292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:16.699267 23292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:16.699330 23292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:53:16.704713 23292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:33381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=34805
--tserver_master_addrs=127.6.152.62:45073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:16.705749 23292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:16.706691 23292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:16.716925 23337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:16.716979 23340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:16.716988 23338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:16.717404 23292 server_base.cc:1034] running on GCE node
I20260928 23:53:16.717837 23292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:16.718568 23292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:16.719745 23292 hybrid_clock.cc:648] HybridClock initialized: now 1790639596719726 us; error 35 us; skew 500 ppm
Sep 28 23:53:16 dist-test-slave-p5bw krb5kdc[22144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639596, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:16.722718 23292 init.cc:377] Logged in from keytab as kudu/127.6.152.1@KRBTEST.COM (short username kudu)
I20260928 23:53:16.723803 23292 webserver.cc:460] Webserver started at http://127.6.152.1:34805/ using document root <none> and password file <none>
I20260928 23:53:16.724491 23292 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:16.724586 23292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:16.729586 23292 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 23:53:16.733770 23347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:16.735306 23292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:53:16.735421 23292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "47fc0f3fa1e649e8ac43fb560b6fd753"
format_stamp: "Formatted at 2026-09-28 23:52:34 on dist-test-slave-p5bw"
server_key: "7b1a85b8d117e3a44d10cccb9f0eb552"
server_key_iv: "1b7c7dc9a6b806fcd3ec615956a36197"
server_key_version: "encryptionkey@0"
I20260928 23:53:16.736122 23292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:16.758116 23292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:16.762250 23292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:16.762468 23292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:16.763257 23292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:53:16.764657 23354 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 23:53:16.768855 23292 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 23:53:16.768939 23292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260928 23:53:16.769008 23292 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 23:53:16.771306 23292 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 23:53:16.771373 23292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 23:53:16.771713 23354 tablet_bootstrap.cc:492] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753: Bootstrap starting.
I20260928 23:53:16.782431 23292 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:33381
I20260928 23:53:16.782823 23461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:33381 every 8 connection(s)
I20260928 23:53:16.783636 23292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:53:16.789455 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 23292
I20260928 23:53:16.789961 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:39253
--local_ip_for_outbound_sockets=127.6.152.2
--tserver_master_addrs=127.6.152.62:45073
--webserver_port=44687
--webserver_interface=127.6.152.2
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:53:16.791567 23354 log.cc:824] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:16.801131 23354 tablet_bootstrap.cc:492] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:53:16.801635 23354 tablet_bootstrap.cc:492] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753: Bootstrap complete.
I20260928 23:53:16.803100 23354 ts_tablet_manager.cc:1397] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753: Time spent bootstrapping tablet: real 0.032s user 0.017s sys 0.012s
I20260928 23:53:16.806264 23354 raft_consensus.cc:348] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:53:16.806783 23354 raft_consensus.cc:729] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47fc0f3fa1e649e8ac43fb560b6fd753, State: Initialized, Role: FOLLOWER
I20260928 23:53:16.807278 23354 consensus_queue.cc:260] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:53:16.808262 23354 ts_tablet_manager.cc:1428] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260928 23:53:16.856340 23464 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:53:16.785913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.1:57953)
0928 23:53:16.786522 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:16.786589 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:16.787711 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:16.855525 (+ 67814us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":141,"threads_started":1}
W20260928 23:53:16.856950 23462 heartbeater.cc:644] Failed to heartbeat to 127.6.152.62:45073 (0 consecutive failures): Network error: Failed to ping master at 127.6.152.62:45073: Client connection negotiation failed: client connection to 127.6.152.62:45073: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 23:53:16.857679 23464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:16.857417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.1:39239)
0928 23:53:16.857545 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:16.857613 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 23:53:16.858385 23464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:16.858177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.1:51215)
0928 23:53:16.858280 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:16.858325 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 23:53:16.858592 23462 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:53:16.908072 23465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:16.908339 23465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:16.908416 23465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:16.912532 23465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:16.912602 23465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:16.912663 23465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:53:16.917937 23465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:39253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=44687
--tserver_master_addrs=127.6.152.62:45073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:16.918808 23465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:16.919740 23465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:16.929127 23475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:16.929140 23473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:16.929136 23472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:16.929314 23465 server_base.cc:1034] running on GCE node
I20260928 23:53:16.929801 23465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:16.930384 23465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:16.931566 23465 hybrid_clock.cc:648] HybridClock initialized: now 1790639596931530 us; error 52 us; skew 500 ppm
Sep 28 23:53:16 dist-test-slave-p5bw krb5kdc[22144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639596, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:16.934271 23465 init.cc:377] Logged in from keytab as kudu/127.6.152.2@KRBTEST.COM (short username kudu)
I20260928 23:53:16.935092 23465 webserver.cc:460] Webserver started at http://127.6.152.2:44687/ using document root <none> and password file <none>
I20260928 23:53:16.935801 23465 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:16.935922 23465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:16.939779 23465 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260928 23:53:16.941814 23482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:16.943058 23465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:53:16.943212 23465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "b5741f7f669e4e22b10ec6b0e1b486be"
format_stamp: "Formatted at 2026-09-28 23:52:34 on dist-test-slave-p5bw"
server_key: "550b514bff59f39c3aac255e6b82b4c2"
server_key_iv: "8123b9a7cacc6653ee97c0544f8cc9e2"
server_key_version: "encryptionkey@0"
I20260928 23:53:16.943768 23465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:16.959453 23465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:16.962762 23465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:16.962968 23465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:16.963575 23465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:53:16.964398 23465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:53:16.964468 23465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:16.964537 23465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:53:16.964579 23465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:16.973363 23465 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:39253
I20260928 23:53:16.973390 23595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:39253 every 8 connection(s)
I20260928 23:53:16.974301 23465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:53:16.977376 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 23465
I20260928 23:53:16.977811 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:45739
--local_ip_for_outbound_sockets=127.6.152.3
--tserver_master_addrs=127.6.152.62:45073
--webserver_port=33547
--webserver_interface=127.6.152.3
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:53:16.978369 23598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:16.976302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.2:44201)
0928 23:53:16.976796 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:16.977039 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":103,"threads_started":1}
W20260928 23:53:16.978884 23596 heartbeater.cc:644] Failed to heartbeat to 127.6.152.62:45073 (0 consecutive failures): Network error: Failed to ping master at 127.6.152.62:45073: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
I20260928 23:53:16.979635 23598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:16.979365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.2:36745)
0928 23:53:16.979532 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:16.979584 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260928 23:53:16.980302 23598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:16.980051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.2:49771)
0928 23:53:16.980176 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:16.980235 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 23:53:16.980491 23596 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:53:17.087014 23599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:17.087265 23599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:17.087311 23599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:17.091180 23599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:17.091228 23599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:17.091284 23599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:53:17.096323 23599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:45739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=33547
--tserver_master_addrs=127.6.152.62:45073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:17.097294 23599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:17.098232 23599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:53:17.107805 23599 server_base.cc:1034] running on GCE node
W20260928 23:53:17.107748 23607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:17.107748 23604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:17.107728 23605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:17.108338 23599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:17.109023 23599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:17.110241 23599 hybrid_clock.cc:648] HybridClock initialized: now 1790639597110219 us; error 30 us; skew 500 ppm
Sep 28 23:53:17 dist-test-slave-p5bw krb5kdc[22144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639597, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:17.112969 23599 init.cc:377] Logged in from keytab as kudu/127.6.152.3@KRBTEST.COM (short username kudu)
I20260928 23:53:17.113804 23599 webserver.cc:460] Webserver started at http://127.6.152.3:33547/ using document root <none> and password file <none>
I20260928 23:53:17.114348 23599 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:17.114442 23599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:17.118242 23599 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 23:53:17.120151 23614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:17.121356 23599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:53:17.121481 23599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "51a1d42c90f7417e94cd5226098d0655"
format_stamp: "Formatted at 2026-09-28 23:52:34 on dist-test-slave-p5bw"
server_key: "134b9a4399ae32ca227e0b3d2389e1f7"
server_key_iv: "0893df6850936f37b4d4bccc89b3ace1"
server_key_version: "encryptionkey@0"
I20260928 23:53:17.122004 23599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:17.145502 23599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:17.148521 23599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:17.148717 23599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:17.149323 23599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:53:17.150116 23599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:53:17.150193 23599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:17.150266 23599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:53:17.150321 23599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:17.159086 23599 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:45739
I20260928 23:53:17.159121 23727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:45739 every 8 connection(s)
I20260928 23:53:17.160187 23599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:53:17.163982 23730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:17.162306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.3:57039)
0928 23:53:17.162734 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:17.162845 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":114,"threads_started":1}
W20260928 23:53:17.164515 23728 heartbeater.cc:644] Failed to heartbeat to 127.6.152.62:45073 (0 consecutive failures): Network error: Failed to ping master at 127.6.152.62:45073: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
I20260928 23:53:17.164777 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 23599
I20260928 23:53:17.165177 23730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:17.164970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.3:42173)
0928 23:53:17.165079 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:17.165125 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 23:53:17.166169 23730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:17.165987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.3:41253)
0928 23:53:17.166082 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:17.166121 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 23:53:17.166384 23728 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 23:53:17.859611 23732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:17.859164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.1:42329)
0928 23:53:17.859449 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:17.859503 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":98,"threads_started":1}
I20260928 23:53:17.981508 23733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:17.981061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.2:38931)
0928 23:53:17.981347 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:17.981407 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":68,"threads_started":1}
I20260928 23:53:18.167397 23734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:18.166928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.3:59735)
0928 23:53:18.167204 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:18.167258 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":102,"threads_started":1}
I20260928 23:53:18.604336 23735 raft_consensus.cc:482] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:53:18.604526 23735 raft_consensus.cc:504] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:53:18.605661 23735 leader_election.cc:288] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51a1d42c90f7417e94cd5226098d0655 (127.6.152.3:45739), b5741f7f669e4e22b10ec6b0e1b486be (127.6.152.2:39253)
Sep 28 23:53:18 dist-test-slave-p5bw krb5kdc[22144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639596, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for kudu/127.6.152.2@KRBTEST.COM
Sep 28 23:53:18 dist-test-slave-p5bw krb5kdc[22144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639596, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for kudu/127.6.152.3@KRBTEST.COM
I20260928 23:53:18.620213 23737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:18.606020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:59231 (local address 127.6.152.3:45739)
0928 23:53:18.606316 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:18.606344 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:18.606392 (+ 48us) server_negotiation.cc:407] Connection header received
0928 23:53:18.607170 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:18.607176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:18.607533 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:18.607845 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:18.609171 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:18.610122 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:18.611627 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:18.612240 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:18.612370 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:18.615805 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:18.615809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:18.615820 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:53:18.615847 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:18.617732 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:18.618269 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:18.618274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:18.618278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:18.618448 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:18.618747 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:18.618749 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:18.618751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:18.619109 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:18.619253 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:18.619778 (+ 525us) server_negotiation.cc:299] Negotiation successful
0928 23:53:18.620024 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":71,"threads_started":1}
I20260928 23:53:18.620210 23738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:18.606209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:39253 (local address 127.6.152.1:38493)
0928 23:53:18.606460 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:18.606480 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:18.606559 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:18.607945 (+ 1386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:18.607948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:18.608468 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:53:18.608997 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:18.609013 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:18.610271 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:18.610275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:18.611334 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:18.611345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:18.612268 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:18.612271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:18.612604 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:18.613366 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:53:18.613395 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:53:18.615172 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:53:18.617697 (+ 2525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:53:18.617703 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:53:18.617718 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:53:18.618053 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:53:18.618435 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:53:18.618438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:53:18.618440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:53:18.618616 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:53:18.619228 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:53:18.619234 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:53:18.619476 (+ 242us) client_negotiation.cc:769] Sending connection context
0928 23:53:18.619661 (+ 185us) client_negotiation.cc:240] Negotiation successful
0928 23:53:18.620004 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":205,"spinlock_wait_cycles":26880,"thread_start_us":50,"threads_started":1}
I20260928 23:53:18.620224 23739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:18.606499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:38493 (local address 127.6.152.2:39253)
0928 23:53:18.606781 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:18.606792 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:18.606839 (+ 47us) server_negotiation.cc:407] Connection header received
0928 23:53:18.607446 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:18.607451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:18.607723 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:18.608006 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:18.609152 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:18.610123 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:18.611476 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:18.612137 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:18.612317 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:18.615441 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:18.615448 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:18.615464 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:53:18.615500 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:18.617576 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:18.618159 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:18.618166 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:18.618174 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:18.618313 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:18.618712 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:18.618715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:18.618717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:18.619101 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:18.619252 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:18.619782 (+ 530us) server_negotiation.cc:299] Negotiation successful
0928 23:53:18.620039 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":92,"threads_started":1}
I20260928 23:53:18.620262 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:18.605866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:45739 (local address 127.6.152.1:59231)
0928 23:53:18.606131 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:18.606149 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:18.606291 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:18.607906 (+ 1615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:18.607912 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:18.608450 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:53:18.608998 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:18.609014 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:18.610267 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:18.610271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:18.611322 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:18.611332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:18.612347 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:18.612351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:18.612513 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:18.613366 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:53:18.613398 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:53:18.615592 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:53:18.617854 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:53:18.617856 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:53:18.617857 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:53:18.618149 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:53:18.618552 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:53:18.618554 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:53:18.618555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:53:18.618628 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:53:18.619228 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:53:18.619234 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:53:18.619475 (+ 241us) client_negotiation.cc:769] Sending connection context
0928 23:53:18.619661 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 23:53:18.620086 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":433,"spinlock_wait_cycles":16640,"thread_start_us":71,"threads_started":1}
W20260928 23:53:18.622972 23348 leader_election.cc:341] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5741f7f669e4e22b10ec6b0e1b486be (127.6.152.2:39253): Not found: Tablet not found: 1633631cb6974f36b3fe5d8be4b8201b
W20260928 23:53:18.623202 23348 leader_election.cc:341] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51a1d42c90f7417e94cd5226098d0655 (127.6.152.3:45739): Not found: Tablet not found: 1633631cb6974f36b3fe5d8be4b8201b
I20260928 23:53:18.623301 23348 leader_election.cc:302] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47fc0f3fa1e649e8ac43fb560b6fd753; no voters: 51a1d42c90f7417e94cd5226098d0655, b5741f7f669e4e22b10ec6b0e1b486be
I20260928 23:53:18.623577 23735 raft_consensus.cc:2726] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:53:18.860759 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:18.860458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.1:33111)
0928 23:53:18.860612 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:18.860675 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 23:53:18.982674 23740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:18.982273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.2:50745)
0928 23:53:18.982521 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:18.982573 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
I20260928 23:53:19.168583 23741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:19.168165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.3:60503)
0928 23:53:19.168438 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:19.168494 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":82,"threads_started":1}
I20260928 23:53:19.861954 23742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:19.861516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.1:54711)
0928 23:53:19.861767 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:19.861830 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":85,"threads_started":1}
I20260928 23:53:19.983814 23743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:19.983388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.2:42823)
0928 23:53:19.983655 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:19.983715 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":72,"threads_started":1}
I20260928 23:53:20.171483 23744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:20.171033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.3:38357)
0928 23:53:20.171307 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:20.171369 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":73,"threads_started":1}
I20260928 23:53:20.694533 23745 raft_consensus.cc:482] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:53:20.694665 23745 raft_consensus.cc:504] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:53:20.695282 23745 leader_election.cc:288] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51a1d42c90f7417e94cd5226098d0655 (127.6.152.3:45739), b5741f7f669e4e22b10ec6b0e1b486be (127.6.152.2:39253)
W20260928 23:53:20.696067 23348 leader_election.cc:341] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51a1d42c90f7417e94cd5226098d0655 (127.6.152.3:45739): Not found: Tablet not found: 1633631cb6974f36b3fe5d8be4b8201b
W20260928 23:53:20.696213 23348 leader_election.cc:341] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5741f7f669e4e22b10ec6b0e1b486be (127.6.152.2:39253): Not found: Tablet not found: 1633631cb6974f36b3fe5d8be4b8201b
I20260928 23:53:20.696270 23348 leader_election.cc:302] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47fc0f3fa1e649e8ac43fb560b6fd753; no voters: 51a1d42c90f7417e94cd5226098d0655, b5741f7f669e4e22b10ec6b0e1b486be
I20260928 23:53:20.696441 23745 raft_consensus.cc:2726] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:53:20.863153 23746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:20.862742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.1:35133)
0928 23:53:20.863006 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:20.863056 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":91,"threads_started":1}
I20260928 23:53:20.985111 23747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:20.984642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.2:60453)
0928 23:53:20.984959 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:20.985017 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":67,"threads_started":1}
I20260928 23:53:21.172652 23748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:21.172242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.3:41001)
0928 23:53:21.172506 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:21.172560 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":88,"threads_started":1}
I20260928 23:53:21.864457 23749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:21.864019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.1:37597)
0928 23:53:21.864288 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:21.864339 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":68,"threads_started":1}
I20260928 23:53:21.986410 23750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:21.985927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.2:39013)
0928 23:53:21.986183 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:21.986282 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":91,"threads_started":1}
I20260928 23:53:22.173879 23751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:22.173484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.3:43373)
0928 23:53:22.173731 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:22.173786 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":72,"threads_started":1}
I20260928 23:53:22.865749 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:22.865316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.1:43551)
0928 23:53:22.865577 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:22.865634 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":91,"threads_started":1}
I20260928 23:53:22.987504 23753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:22.987098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.2:40975)
0928 23:53:22.987355 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:22.987412 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":69,"threads_started":1}
I20260928 23:53:23.175154 23754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:23.174696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.3:52847)
0928 23:53:23.174992 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:23.175054 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":85,"threads_started":1}
I20260928 23:53:23.396979 23755 raft_consensus.cc:482] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:53:23.397140 23755 raft_consensus.cc:504] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:53:23.397543 23755 leader_election.cc:288] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51a1d42c90f7417e94cd5226098d0655 (127.6.152.3:45739), b5741f7f669e4e22b10ec6b0e1b486be (127.6.152.2:39253)
W20260928 23:53:23.398492 23348 leader_election.cc:341] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51a1d42c90f7417e94cd5226098d0655 (127.6.152.3:45739): Not found: Tablet not found: 1633631cb6974f36b3fe5d8be4b8201b
W20260928 23:53:23.398636 23348 leader_election.cc:341] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5741f7f669e4e22b10ec6b0e1b486be (127.6.152.2:39253): Not found: Tablet not found: 1633631cb6974f36b3fe5d8be4b8201b
I20260928 23:53:23.398697 23348 leader_election.cc:302] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47fc0f3fa1e649e8ac43fb560b6fd753; no voters: 51a1d42c90f7417e94cd5226098d0655, b5741f7f669e4e22b10ec6b0e1b486be
I20260928 23:53:23.398882 23755 raft_consensus.cc:2726] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:53:23.866918 23756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:23.866529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.1:49067)
0928 23:53:23.866762 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:23.866818 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260928 23:53:23.989105 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:23.988299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.2:58289)
0928 23:53:23.988772 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:23.988959 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":103,"threads_started":1}
I20260928 23:53:24.176388 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:24.175963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.3:49663)
0928 23:53:24.176245 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:24.176295 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":98,"threads_started":1}
I20260928 23:53:24.868192 23759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:24.867747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.1:43757)
0928 23:53:24.868031 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:24.868095 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":132,"threads_started":1}
I20260928 23:53:24.990222 23760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:24.989828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.2:59687)
0928 23:53:24.990073 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:24.990125 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":73,"threads_started":1}
I20260928 23:53:25.177569 23761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:25.177148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.3:34023)
0928 23:53:25.177415 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:25.177474 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
I20260928 23:53:25.296357 23762 raft_consensus.cc:482] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:53:25.296518 23762 raft_consensus.cc:504] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:53:25.296936 23762 leader_election.cc:288] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51a1d42c90f7417e94cd5226098d0655 (127.6.152.3:45739), b5741f7f669e4e22b10ec6b0e1b486be (127.6.152.2:39253)
W20260928 23:53:25.297685 23348 leader_election.cc:341] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51a1d42c90f7417e94cd5226098d0655 (127.6.152.3:45739): Not found: Tablet not found: 1633631cb6974f36b3fe5d8be4b8201b
W20260928 23:53:25.297833 23348 leader_election.cc:341] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5741f7f669e4e22b10ec6b0e1b486be (127.6.152.2:39253): Not found: Tablet not found: 1633631cb6974f36b3fe5d8be4b8201b
I20260928 23:53:25.297900 23348 leader_election.cc:302] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47fc0f3fa1e649e8ac43fb560b6fd753; no voters: 51a1d42c90f7417e94cd5226098d0655, b5741f7f669e4e22b10ec6b0e1b486be
I20260928 23:53:25.298079 23762 raft_consensus.cc:2726] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:53:25.869442 23763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:25.869027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.1:51105)
0928 23:53:25.869278 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:25.869330 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":72,"threads_started":1}
I20260928 23:53:25.991309 23764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:25.990927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.2:35109)
0928 23:53:25.991167 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:25.991218 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
I20260928 23:53:26.178649 23765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:26.178273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.3:57665)
0928 23:53:26.178507 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:26.178560 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":85,"threads_started":1}
I20260928 23:53:26.870801 23766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:26.870355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.1:52925)
0928 23:53:26.870639 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:26.870703 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":86,"threads_started":1}
I20260928 23:53:26.992491 23767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:26.992097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.2:52803)
0928 23:53:26.992343 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:26.992397 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":80,"threads_started":1}
I20260928 23:53:27.179929 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:27.179492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.3:48365)
0928 23:53:27.179754 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:27.179816 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
I20260928 23:53:27.872165 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:27.871638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.1:41797)
0928 23:53:27.871940 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:27.872021 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":67,"threads_started":1}
I20260928 23:53:27.993848 23770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:27.993374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.2:47673)
0928 23:53:27.993651 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:27.993716 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
I20260928 23:53:28.181214 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:28.180765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.3:56225)
0928 23:53:28.181069 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:28.181121 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":135,"threads_started":1}
I20260928 23:53:28.746502 23772 raft_consensus.cc:482] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:53:28.746711 23772 raft_consensus.cc:504] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:53:28.747327 23772 leader_election.cc:288] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51a1d42c90f7417e94cd5226098d0655 (127.6.152.3:45739), b5741f7f669e4e22b10ec6b0e1b486be (127.6.152.2:39253)
W20260928 23:53:28.748099 23348 leader_election.cc:341] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51a1d42c90f7417e94cd5226098d0655 (127.6.152.3:45739): Not found: Tablet not found: 1633631cb6974f36b3fe5d8be4b8201b
W20260928 23:53:28.748250 23348 leader_election.cc:341] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5741f7f669e4e22b10ec6b0e1b486be (127.6.152.2:39253): Not found: Tablet not found: 1633631cb6974f36b3fe5d8be4b8201b
I20260928 23:53:28.748299 23348 leader_election.cc:302] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47fc0f3fa1e649e8ac43fb560b6fd753; no voters: 51a1d42c90f7417e94cd5226098d0655, b5741f7f669e4e22b10ec6b0e1b486be
I20260928 23:53:28.748458 23772 raft_consensus.cc:2726] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:53:28.873452 23773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:28.872982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.1:58439)
0928 23:53:28.873290 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:28.873349 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":111,"threads_started":1}
I20260928 23:53:28.995112 23774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:28.994689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.2:40637)
0928 23:53:28.994954 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:28.995013 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":78,"threads_started":1}
I20260928 23:53:29.182399 23775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:29.181977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.3:46201)
0928 23:53:29.182250 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:29.182302 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":111,"threads_started":1}
I20260928 23:53:29.874778 23776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:29.874270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.1:52571)
0928 23:53:29.874582 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:29.874645 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":105,"threads_started":1}
I20260928 23:53:29.996223 23777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:29.995833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.2:37943)
0928 23:53:29.996077 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:29.996128 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":69,"threads_started":1}
I20260928 23:53:30.183519 23778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:30.183098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.3:58121)
0928 23:53:30.183366 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:30.183424 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":84,"threads_started":1}
I20260928 23:53:30.876185 23779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:30.875770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.1:43303)
0928 23:53:30.876029 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:30.876087 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":89,"threads_started":1}
I20260928 23:53:30.973245 23780 raft_consensus.cc:482] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:53:30.973382 23780 raft_consensus.cc:504] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a1d42c90f7417e94cd5226098d0655" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 45739 } } peers { permanent_uuid: "b5741f7f669e4e22b10ec6b0e1b486be" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 39253 } } peers { permanent_uuid: "47fc0f3fa1e649e8ac43fb560b6fd753" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 33381 } }
I20260928 23:53:30.973714 23780 leader_election.cc:288] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51a1d42c90f7417e94cd5226098d0655 (127.6.152.3:45739), b5741f7f669e4e22b10ec6b0e1b486be (127.6.152.2:39253)
W20260928 23:53:30.974464 23348 leader_election.cc:341] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51a1d42c90f7417e94cd5226098d0655 (127.6.152.3:45739): Not found: Tablet not found: 1633631cb6974f36b3fe5d8be4b8201b
W20260928 23:53:30.974644 23348 leader_election.cc:341] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5741f7f669e4e22b10ec6b0e1b486be (127.6.152.2:39253): Not found: Tablet not found: 1633631cb6974f36b3fe5d8be4b8201b
I20260928 23:53:30.974702 23348 leader_election.cc:302] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47fc0f3fa1e649e8ac43fb560b6fd753; no voters: 51a1d42c90f7417e94cd5226098d0655, b5741f7f669e4e22b10ec6b0e1b486be
I20260928 23:53:30.974833 23780 raft_consensus.cc:2726] T 1633631cb6974f36b3fe5d8be4b8201b P 47fc0f3fa1e649e8ac43fb560b6fd753 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:53:30.997318 23781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:30.996932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.2:40059)
0928 23:53:30.997170 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:30.997223 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":83,"threads_started":1}
I20260928 23:53:31.184619 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:31.184207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.3:34649)
0928 23:53:31.184464 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:31.184525 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":76,"threads_started":1}
I20260928 23:53:31.877808 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:31.877326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.1:59641)
0928 23:53:31.877610 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:31.877670 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":116,"threads_started":1}
I20260928 23:53:31.998597 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:31.998227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45073 (local address 127.6.152.2:49633)
0928 23:53:31.998455 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:31.998504 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.62:45073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":67,"threads_started":1}
I20260928 23:53:32.171437 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 23292
I20260928 23:53:32.177054 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 23465
I20260928 23:53:32.182152 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 23599
2026-09-28T23:53:32Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59356 ms)
[----------] 30 tests from SecurityITest (237628 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:53:32 dist-test-slave-p5bw krb5kdc[23787](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:53:32 dist-test-slave-p5bw krb5kdc[23787](info): set up 2 sockets
Sep 28 23:53:32 dist-test-slave-p5bw krb5kdc[23787](info): commencing operation
krb5kdc: starting...
W20260928 23:53:33.213279 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:53:33 dist-test-slave-p5bw krb5kdc[23787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:53:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:53:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:53:33.335744 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:46481
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:46481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:53:33.446177 23800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:33.446431 23800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:33.446472 23800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:33.450332 23800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:53:33.450387 23800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:33.450405 23800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:33.450423 23800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:53:33.450443 23800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:53:33.455574 23800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:46481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:46481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:33.456519 23800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:33.457572 23800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:33.466157 23806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:33.466241 23808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:33.466315 23800 server_base.cc:1034] running on GCE node
W20260928 23:53:33.466245 23805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:33.466970 23800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:33.468303 23800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:33.469519 23800 hybrid_clock.cc:648] HybridClock initialized: now 1790639613469500 us; error 32 us; skew 500 ppm
Sep 28 23:53:33 dist-test-slave-p5bw krb5kdc[23787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639613, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:33.479007 23800 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:53:33.480080 23800 webserver.cc:460] Webserver started at http://127.6.152.62:33185/ using document root <none> and password file <none>
I20260928 23:53:33.480674 23800 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:33.480743 23800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:33.481003 23800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:33.483259 23800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "c9e106c46c674e80aa464f6089a3ce45"
format_stamp: "Formatted at 2026-09-28 23:53:33 on dist-test-slave-p5bw"
server_key: "07aec669e278b17b0fc2c630ae34f9b5"
server_key_iv: "cd081a83048045c59cca97a7c3d2c0f8"
server_key_version: "encryptionkey@0"
I20260928 23:53:33.483826 23800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "c9e106c46c674e80aa464f6089a3ce45"
format_stamp: "Formatted at 2026-09-28 23:53:33 on dist-test-slave-p5bw"
server_key: "07aec669e278b17b0fc2c630ae34f9b5"
server_key_iv: "cd081a83048045c59cca97a7c3d2c0f8"
server_key_version: "encryptionkey@0"
I20260928 23:53:33.487885 23800 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 23:53:33.490595 23815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:33.491879 23800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:53:33.491995 23800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "c9e106c46c674e80aa464f6089a3ce45"
format_stamp: "Formatted at 2026-09-28 23:53:33 on dist-test-slave-p5bw"
server_key: "07aec669e278b17b0fc2c630ae34f9b5"
server_key_iv: "cd081a83048045c59cca97a7c3d2c0f8"
server_key_version: "encryptionkey@0"
I20260928 23:53:33.492084 23800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:33.517730 23800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:33.520589 23800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:33.520745 23800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:33.527626 23800 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:46481
I20260928 23:53:33.527640 23867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:46481 every 8 connection(s)
I20260928 23:53:33.528559 23800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:53:33.531126 23868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:33.532047 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 23800
I20260928 23:53:33.532182 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:53:33.532506 6752 external_mini_cluster.cc:1428] Setting key 2d84ec43c8529b5125e8ec1a841ed39f
I20260928 23:53:33.537590 23868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9e106c46c674e80aa464f6089a3ce45: Bootstrap starting.
I20260928 23:53:33.539999 23868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9e106c46c674e80aa464f6089a3ce45: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:33.540696 23868 log.cc:824] T 00000000000000000000000000000000 P c9e106c46c674e80aa464f6089a3ce45: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:53:33 dist-test-slave-p5bw krb5kdc[23787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:53:33.542814 23868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9e106c46c674e80aa464f6089a3ce45: No bootstrap required, opened a new log
I20260928 23:53:33.547386 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:33.533895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47546 (local address 127.6.152.62:46481)
0928 23:53:33.534293 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:33.534315 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:33.534360 (+ 45us) server_negotiation.cc:407] Connection header received
0928 23:53:33.535145 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:33.535167 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:33.535565 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:33.535937 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:33.536550 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:33.537983 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:33.538794 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:33.539444 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:33.539602 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:33.542110 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:33.542116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:33.542131 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:53:33.542158 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:33.544100 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:33.544489 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:33.544494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:33.544500 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:33.544656 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:33.545006 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:33.545010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:33.545013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:33.545644 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:33.545826 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:33.546081 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 23:53:33.546398 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":100,"threads_started":1}
I20260928 23:53:33.547412 23868 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9e106c46c674e80aa464f6089a3ce45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e106c46c674e80aa464f6089a3ce45" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 46481 } }
I20260928 23:53:33.547756 23868 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9e106c46c674e80aa464f6089a3ce45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:33.547825 23868 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9e106c46c674e80aa464f6089a3ce45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e106c46c674e80aa464f6089a3ce45, State: Initialized, Role: FOLLOWER
I20260928 23:53:33.548219 23868 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9e106c46c674e80aa464f6089a3ce45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e106c46c674e80aa464f6089a3ce45" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 46481 } }
I20260928 23:53:33.548370 23868 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9e106c46c674e80aa464f6089a3ce45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:53:33.548460 23868 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9e106c46c674e80aa464f6089a3ce45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:53:33.548561 23868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9e106c46c674e80aa464f6089a3ce45 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:33.549667 23868 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9e106c46c674e80aa464f6089a3ce45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e106c46c674e80aa464f6089a3ce45" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 46481 } }
I20260928 23:53:33.550016 23868 leader_election.cc:302] T 00000000000000000000000000000000 P c9e106c46c674e80aa464f6089a3ce45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9e106c46c674e80aa464f6089a3ce45; no voters:
I20260928 23:53:33.550328 23868 leader_election.cc:288] T 00000000000000000000000000000000 P c9e106c46c674e80aa464f6089a3ce45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:53:33.550438 23873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9e106c46c674e80aa464f6089a3ce45 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:53:33.550655 23873 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9e106c46c674e80aa464f6089a3ce45 [term 1 LEADER]: Becoming Leader. State: Replica: c9e106c46c674e80aa464f6089a3ce45, State: Running, Role: LEADER
I20260928 23:53:33.550976 23873 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9e106c46c674e80aa464f6089a3ce45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e106c46c674e80aa464f6089a3ce45" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 46481 } }
I20260928 23:53:33.551587 23868 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9e106c46c674e80aa464f6089a3ce45 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:53:33.552652 23874 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9e106c46c674e80aa464f6089a3ce45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9e106c46c674e80aa464f6089a3ce45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e106c46c674e80aa464f6089a3ce45" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 46481 } } }
I20260928 23:53:33.552778 23874 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9e106c46c674e80aa464f6089a3ce45 [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:33.553022 23875 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9e106c46c674e80aa464f6089a3ce45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9e106c46c674e80aa464f6089a3ce45. Latest consensus state: current_term: 1 leader_uuid: "c9e106c46c674e80aa464f6089a3ce45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e106c46c674e80aa464f6089a3ce45" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 46481 } } }
I20260928 23:53:33.553117 23875 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9e106c46c674e80aa464f6089a3ce45 [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:33.553368 23882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:53:33.555887 23882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:53:33.560953 23882 catalog_manager.cc:1347] Generated new cluster ID: 24c3d06661f04523b0c46438dd694619
I20260928 23:53:33.561012 23882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:53:33.573460 23882 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:53:33.574687 23882 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:53:33.579348 23882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9e106c46c674e80aa464f6089a3ce45: Generated new TSK 0
I20260928 23:53:33.580108 23882 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
I20260928 23:53:33.607980 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:46481
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:53:33.716869 23894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:33.717111 23894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:33.717178 23894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:33.721189 23894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:33.721247 23894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:33.721349 23894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:53:33.726436 23894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:46481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:33.727456 23894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:33.728446 23894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:33.737643 23902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:33.737658 23900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:33.737632 23899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:33.737905 23894 server_base.cc:1034] running on GCE node
I20260928 23:53:33.738391 23894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:33.739112 23894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:33.740314 23894 hybrid_clock.cc:648] HybridClock initialized: now 1790639613740298 us; error 37 us; skew 500 ppm
Sep 28 23:53:33 dist-test-slave-p5bw krb5kdc[23787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639613, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:33.743048 23894 init.cc:377] Logged in from keytab as kudu/127.6.152.1@KRBTEST.COM (short username kudu)
I20260928 23:53:33.743906 23894 webserver.cc:460] Webserver started at http://127.6.152.1:36475/ using document root <none> and password file <none>
I20260928 23:53:33.744459 23894 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:33.744545 23894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:33.744776 23894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:33.746829 23894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "7d8d20c28b4446f3a3063914c718ec3c"
format_stamp: "Formatted at 2026-09-28 23:53:33 on dist-test-slave-p5bw"
server_key: "53378f408e3b076922981b0a21c80359"
server_key_iv: "56c4a1448b414b1f30bf8930a6f7f3cf"
server_key_version: "encryptionkey@0"
I20260928 23:53:33.747387 23894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "7d8d20c28b4446f3a3063914c718ec3c"
format_stamp: "Formatted at 2026-09-28 23:53:33 on dist-test-slave-p5bw"
server_key: "53378f408e3b076922981b0a21c80359"
server_key_iv: "56c4a1448b414b1f30bf8930a6f7f3cf"
server_key_version: "encryptionkey@0"
I20260928 23:53:33.751003 23894 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260928 23:53:33.753697 23909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:33.754905 23894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260928 23:53:33.755031 23894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "7d8d20c28b4446f3a3063914c718ec3c"
format_stamp: "Formatted at 2026-09-28 23:53:33 on dist-test-slave-p5bw"
server_key: "53378f408e3b076922981b0a21c80359"
server_key_iv: "56c4a1448b414b1f30bf8930a6f7f3cf"
server_key_version: "encryptionkey@0"
I20260928 23:53:33.755136 23894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:33.770363 23894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:33.773316 23894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:33.773504 23894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:33.774061 23894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:53:33.774847 23894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:53:33.774910 23894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:33.774984 23894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:53:33.775024 23894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:33.783234 23894 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:42081
I20260928 23:53:33.783288 24022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:42081 every 8 connection(s)
I20260928 23:53:33.784107 23894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
Sep 28 23:53:33 dist-test-slave-p5bw krb5kdc[23787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639613, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:53:33.793972 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 23894
I20260928 23:53:33.794137 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
I20260928 23:53:33.794436 6752 external_mini_cluster.cc:1428] Setting key 791da56aa4112d4308b231200be22973
I20260928 23:53:33.797523 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:33.785841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:60129 (local address 127.6.152.62:46481)
0928 23:53:33.786003 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:33.786006 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:33.786841 (+ 835us) server_negotiation.cc:407] Connection header received
0928 23:53:33.787760 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:33.787764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:33.787821 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:33.787918 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:33.789218 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:33.789761 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:33.790902 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:33.791136 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:33.791444 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:33.793757 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:33.793761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:33.793763 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:53:33.793787 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:33.795395 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:33.795906 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:33.795910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:33.795913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:33.796006 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:33.796392 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:33.796395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:33.796396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:33.796567 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:33.796643 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:33.797294 (+ 651us) server_negotiation.cc:299] Negotiation successful
0928 23:53:33.797406 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 23:53:33.798447 24025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:33.786209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:46481 (local address 127.6.152.1:60129)
0928 23:53:33.786669 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:33.786726 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:33.787545 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:33.788093 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:33.788099 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:33.788498 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:53:33.789065 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:33.789081 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:33.789893 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:33.789896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:33.790778 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:33.790787 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:33.791474 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:33.791478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:33.791624 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:33.792167 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:53:33.792181 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:53:33.793583 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:53:33.795494 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:53:33.795499 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:53:33.795511 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:53:33.795810 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:53:33.796106 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:53:33.796109 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:53:33.796111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:53:33.796294 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:53:33.796667 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:53:33.796677 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:53:33.797051 (+ 374us) client_negotiation.cc:769] Sending connection context
0928 23:53:33.797263 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 23:53:33.797621 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":120,"threads_started":1}
I20260928 23:53:33.799618 24023 heartbeater.cc:347] Connected to a master server at 127.6.152.62:46481
I20260928 23:53:33.799849 24023 heartbeater.cc:464] Registering TS with master...
I20260928 23:53:33.800369 24023 heartbeater.cc:511] Master 127.6.152.62:46481 requested a full tablet report, sending...
I20260928 23:53:33.801836 23832 ts_manager.cc:194] Registered new tserver with Master: 7d8d20c28b4446f3a3063914c718ec3c (127.6.152.1:42081)
I20260928 23:53:33.802982 23832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.1@KRBTEST.COM'} at 127.6.152.1:60129
WARNING: no policy specified for kudu/127.6.152.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
I20260928 23:53:33.827255 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:46481
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:53:33.938730 24028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:33.939047 24028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:33.939131 24028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:33.944444 24028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:33.944528 24028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:33.944625 24028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:53:33.950130 24028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:46481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:33.951222 24028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:33.952302 24028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:33.962163 24036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:33.962206 24033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:33.962353 24028 server_base.cc:1034] running on GCE node
W20260928 23:53:33.962505 24034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:33.962962 24028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:33.963670 24028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:33.964881 24028 hybrid_clock.cc:648] HybridClock initialized: now 1790639613964857 us; error 30 us; skew 500 ppm
Sep 28 23:53:33 dist-test-slave-p5bw krb5kdc[23787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639613, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:33.967527 24028 init.cc:377] Logged in from keytab as kudu/127.6.152.2@KRBTEST.COM (short username kudu)
I20260928 23:53:33.968426 24028 webserver.cc:460] Webserver started at http://127.6.152.2:40065/ using document root <none> and password file <none>
I20260928 23:53:33.969046 24028 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:33.969132 24028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:33.969352 24028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:33.971329 24028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "fbe8990f78b34e8d8900ec9a243fee74"
format_stamp: "Formatted at 2026-09-28 23:53:33 on dist-test-slave-p5bw"
server_key: "60372e222b3c1c9a8d64bb33860c4cb8"
server_key_iv: "4d489a24a83dcd770f3d6af922aa360e"
server_key_version: "encryptionkey@0"
I20260928 23:53:33.971885 24028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "fbe8990f78b34e8d8900ec9a243fee74"
format_stamp: "Formatted at 2026-09-28 23:53:33 on dist-test-slave-p5bw"
server_key: "60372e222b3c1c9a8d64bb33860c4cb8"
server_key_iv: "4d489a24a83dcd770f3d6af922aa360e"
server_key_version: "encryptionkey@0"
I20260928 23:53:33.975580 24028 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 23:53:33.977907 24043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:33.979019 24028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:53:33.979139 24028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "fbe8990f78b34e8d8900ec9a243fee74"
format_stamp: "Formatted at 2026-09-28 23:53:33 on dist-test-slave-p5bw"
server_key: "60372e222b3c1c9a8d64bb33860c4cb8"
server_key_iv: "4d489a24a83dcd770f3d6af922aa360e"
server_key_version: "encryptionkey@0"
I20260928 23:53:33.979240 24028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:33.988427 24028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:33.991215 24028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:33.991405 24028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:33.991930 24028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:53:33.992697 24028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:53:33.992759 24028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:33.992844 24028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:53:33.992899 24028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:34.001585 24028 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:32785
I20260928 23:53:34.001616 24156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:32785 every 8 connection(s)
I20260928 23:53:34.002413 24028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:53:34.003504 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 24028
I20260928 23:53:34.003630 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:53:34.003988 6752 external_mini_cluster.cc:1428] Setting key 4a1d0408011636b0a74e9119ac266692
Sep 28 23:53:34 dist-test-slave-p5bw krb5kdc[23787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639613, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.152.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.3@KRBTEST.COM" created.
I20260928 23:53:34.017501 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:34.004166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:38153 (local address 127.6.152.62:46481)
0928 23:53:34.004298 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:34.004301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:34.005218 (+ 917us) server_negotiation.cc:407] Connection header received
0928 23:53:34.006403 (+ 1185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:34.006406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:34.006449 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:34.006512 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:34.007934 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.008697 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.010641 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.010850 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.010935 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:34.013769 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:34.013772 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:34.013774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:53:34.013796 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:34.015317 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:34.015818 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:34.015821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:34.015823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:34.015899 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:34.016252 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:34.016258 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:34.016262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:34.016454 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:34.016552 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:34.017179 (+ 627us) server_negotiation.cc:299] Negotiation successful
0928 23:53:34.017311 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:53:34.018309 24159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:34.004571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:46481 (local address 127.6.152.2:38153)
0928 23:53:34.005054 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:34.005107 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:34.006168 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:34.006682 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:34.006693 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:34.007227 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:53:34.007787 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:34.007801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.008948 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.008955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:34.010521 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:34.010534 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.011010 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.011017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:34.011274 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:34.011894 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:53:34.011921 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:53:34.013595 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:53:34.015445 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:53:34.015451 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:53:34.015465 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:53:34.015714 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:53:34.015986 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:53:34.015990 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:53:34.015992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:53:34.016153 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:53:34.016562 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:53:34.016570 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:53:34.016907 (+ 337us) client_negotiation.cc:769] Sending connection context
0928 23:53:34.017125 (+ 218us) client_negotiation.cc:240] Negotiation successful
0928 23:53:34.017467 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":108,"threads_started":1}
I20260928 23:53:34.019620 24157 heartbeater.cc:347] Connected to a master server at 127.6.152.62:46481
I20260928 23:53:34.019877 24157 heartbeater.cc:464] Registering TS with master...
I20260928 23:53:34.020452 24157 heartbeater.cc:511] Master 127.6.152.62:46481 requested a full tablet report, sending...
I20260928 23:53:34.021607 23832 ts_manager.cc:194] Registered new tserver with Master: fbe8990f78b34e8d8900ec9a243fee74 (127.6.152.2:32785)
I20260928 23:53:34.022181 23832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.2@KRBTEST.COM'} at 127.6.152.2:38153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
I20260928 23:53:34.034013 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:46481
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:53:34.154702 24162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:34.155012 24162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:34.155109 24162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:34.159919 24162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:34.160002 24162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:34.160100 24162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:53:34.165735 24162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:46481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:34.166808 24162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:34.167829 24162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:34.177136 24168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:34.177145 24170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:34.177145 24167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:34.177296 24162 server_base.cc:1034] running on GCE node
I20260928 23:53:34.177788 24162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:34.178393 24162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:34.179566 24162 hybrid_clock.cc:648] HybridClock initialized: now 1790639614179539 us; error 49 us; skew 500 ppm
Sep 28 23:53:34 dist-test-slave-p5bw krb5kdc[23787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639614, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:34.182327 24162 init.cc:377] Logged in from keytab as kudu/127.6.152.3@KRBTEST.COM (short username kudu)
I20260928 23:53:34.183221 24162 webserver.cc:460] Webserver started at http://127.6.152.3:33179/ using document root <none> and password file <none>
I20260928 23:53:34.183760 24162 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:34.183841 24162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:34.184057 24162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:34.186045 24162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "13a24e940bbc4463a0244bf4e4c96cbc"
format_stamp: "Formatted at 2026-09-28 23:53:34 on dist-test-slave-p5bw"
server_key: "cbfdfcf3d97273cf0863758803c7ecf5"
server_key_iv: "43d37ab8e25061157a76cd2b83922a85"
server_key_version: "encryptionkey@0"
I20260928 23:53:34.186582 24162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "13a24e940bbc4463a0244bf4e4c96cbc"
format_stamp: "Formatted at 2026-09-28 23:53:34 on dist-test-slave-p5bw"
server_key: "cbfdfcf3d97273cf0863758803c7ecf5"
server_key_iv: "43d37ab8e25061157a76cd2b83922a85"
server_key_version: "encryptionkey@0"
I20260928 23:53:34.190151 24162 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:53:34.192445 24177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:34.193655 24162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:53:34.193775 24162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "13a24e940bbc4463a0244bf4e4c96cbc"
format_stamp: "Formatted at 2026-09-28 23:53:34 on dist-test-slave-p5bw"
server_key: "cbfdfcf3d97273cf0863758803c7ecf5"
server_key_iv: "43d37ab8e25061157a76cd2b83922a85"
server_key_version: "encryptionkey@0"
I20260928 23:53:34.193876 24162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:34.206542 24162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:34.210129 24162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:34.210342 24162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:34.210953 24162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:53:34.211784 24162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:53:34.211850 24162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:34.211922 24162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:53:34.211962 24162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:34.221127 24162 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:46809
I20260928 23:53:34.221153 24290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:46809 every 8 connection(s)
I20260928 23:53:34.222054 24162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:53:34.230191 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 24162
I20260928 23:53:34.230348 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:53:34.230683 6752 external_mini_cluster.cc:1428] Setting key e1d7d6d9f35859e522495fa229edc6df
Sep 28 23:53:34 dist-test-slave-p5bw krb5kdc[23787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639614, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:53:34.235790 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:34.223960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:40385 (local address 127.6.152.62:46481)
0928 23:53:34.224109 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:34.224113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:34.225031 (+ 918us) server_negotiation.cc:407] Connection header received
0928 23:53:34.225937 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:34.225941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:34.226000 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:34.226100 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:34.227308 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.227839 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.229037 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.229261 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.229381 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:34.232412 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:34.232415 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:34.232417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:53:34.232441 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:34.233950 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:34.234454 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:34.234456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:34.234458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:34.234536 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:34.234826 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:34.234828 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:34.234830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:34.235006 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:34.235085 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:34.235546 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 23:53:34.235658 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 23:53:34.236615 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:34.224350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:46481 (local address 127.6.152.3:40385)
0928 23:53:34.224876 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:34.224924 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:34.225719 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:34.226272 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:34.226279 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:34.226649 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:53:34.227145 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:34.227159 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.227967 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.227971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:34.228909 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:34.228918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.229421 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.229426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:34.229597 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:34.230244 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:53:34.230267 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:53:34.232217 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:53:34.234055 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:53:34.234060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:53:34.234072 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:53:34.234362 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:53:34.234614 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:53:34.234616 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:53:34.234617 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:53:34.234744 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:53:34.235089 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:53:34.235093 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:53:34.235341 (+ 248us) client_negotiation.cc:769] Sending connection context
0928 23:53:34.235496 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 23:53:34.235736 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":108,"threads_started":1}
I20260928 23:53:34.237722 24291 heartbeater.cc:347] Connected to a master server at 127.6.152.62:46481
I20260928 23:53:34.237936 24291 heartbeater.cc:464] Registering TS with master...
I20260928 23:53:34.238359 24291 heartbeater.cc:511] Master 127.6.152.62:46481 requested a full tablet report, sending...
I20260928 23:53:34.239311 23832 ts_manager.cc:194] Registered new tserver with Master: 13a24e940bbc4463a0244bf4e4c96cbc (127.6.152.3:46809)
I20260928 23:53:34.239890 23832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.3@KRBTEST.COM'} at 127.6.152.3:40385
I20260928 23:53:34.244622 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:53:34.253491 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:34.246544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47548 (local address 127.6.152.62:46481)
0928 23:53:34.246663 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:34.246666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:34.246835 (+ 169us) server_negotiation.cc:407] Connection header received
0928 23:53:34.246928 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:34.246930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:34.246965 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:34.247029 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:34.247623 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.248116 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.249032 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.249241 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.249325 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:34.250510 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:34.250513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:34.250515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:53:34.250535 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:34.251978 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:34.252374 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:34.252377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:34.252378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:34.252450 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:34.252646 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:34.252648 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:34.252649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:34.252896 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:34.253021 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:34.253266 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 23:53:34.253379 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:53:34.256003 23832 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:53:34.259214 23832 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:53:34.272744 24307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:34.267995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51988 (local address 127.6.152.3:46809)
0928 23:53:34.268446 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:34.268461 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:34.268485 (+ 24us) server_negotiation.cc:407] Connection header received
0928 23:53:34.268565 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:34.268574 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:34.268791 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:34.268977 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:34.269253 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.270118 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.271550 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.272087 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.272208 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:34.272372 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:34.272490 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 23:53:34.272580 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":134,"threads_started":1}
I20260928 23:53:34.272738 24303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:34.267368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:46809 (local address 127.0.0.1:51988)
0928 23:53:34.268109 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:34.268161 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:34.268319 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:34.268963 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:34.268969 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:34.268990 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:34.269136 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:34.269145 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.270238 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.270243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:34.271437 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:34.271446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.272206 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.272210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:34.272315 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:34.272350 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 23:53:34.272461 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 23:53:34.272580 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":9,"thread_start_us":94,"threads_started":1}
I20260928 23:53:34.273707 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:34.267200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:42081 (local address 127.0.0.1:59510)
0928 23:53:34.268308 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:34.268321 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:34.268475 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:34.269170 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:34.269175 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:34.269197 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:34.269276 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:34.269280 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.270669 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.270673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:34.272095 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:34.272106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.273353 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.273357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:34.273450 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:34.273471 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 23:53:34.273541 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 23:53:34.273590 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":84,"thread_start_us":65,"threads_started":1}
I20260928 23:53:34.273834 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:34.267368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59510 (local address 127.6.152.1:42081)
0928 23:53:34.267746 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:34.267761 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:34.268408 (+ 647us) server_negotiation.cc:407] Connection header received
0928 23:53:34.268623 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:34.268629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:34.268868 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:34.268991 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:34.269381 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.270519 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.272233 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.273088 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.273178 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:34.273313 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:34.273581 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 23:53:34.273675 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":99,"threads_started":1}
I20260928 23:53:34.274533 24302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:34.267666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:32785 (local address 127.0.0.1:56512)
0928 23:53:34.268544 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:34.268557 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:34.268621 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:34.269359 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:34.269363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:34.269380 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:34.269478 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:34.269484 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.270822 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.270825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:34.272095 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:34.272106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.274157 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.274161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:34.274259 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:34.274282 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:53:34.274352 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 23:53:34.274416 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":77,"thread_start_us":97,"threads_started":1}
I20260928 23:53:34.275022 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:34.267945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56512 (local address 127.6.152.2:32785)
0928 23:53:34.268739 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:34.268750 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:34.268770 (+ 20us) server_negotiation.cc:407] Connection header received
0928 23:53:34.268869 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:34.268879 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:34.269046 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:34.269181 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:34.269600 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.270672 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.272233 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.273848 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.273960 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:34.274094 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:34.274788 (+ 694us) server_negotiation.cc:299] Negotiation successful
0928 23:53:34.274884 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":147,"threads_started":1}
I20260928 23:53:34.275902 24225 tablet_service.cc:1467] Processing CreateTablet for tablet 88049927697445e981ed7096ae0005c1 (DEFAULT_TABLE table=test-table [id=ad501de2b89946e9ad97482040ff655b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:53:34.276616 23957 tablet_service.cc:1467] Processing CreateTablet for tablet 88049927697445e981ed7096ae0005c1 (DEFAULT_TABLE table=test-table [id=ad501de2b89946e9ad97482040ff655b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:53:34.277078 24225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88049927697445e981ed7096ae0005c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:34.277518 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88049927697445e981ed7096ae0005c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:34.277933 24091 tablet_service.cc:1467] Processing CreateTablet for tablet 88049927697445e981ed7096ae0005c1 (DEFAULT_TABLE table=test-table [id=ad501de2b89946e9ad97482040ff655b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:53:34.278921 24091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88049927697445e981ed7096ae0005c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:34.284152 24308 tablet_bootstrap.cc:492] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c: Bootstrap starting.
I20260928 23:53:34.285768 24310 tablet_bootstrap.cc:492] T 88049927697445e981ed7096ae0005c1 P fbe8990f78b34e8d8900ec9a243fee74: Bootstrap starting.
I20260928 23:53:34.286384 24309 tablet_bootstrap.cc:492] T 88049927697445e981ed7096ae0005c1 P 13a24e940bbc4463a0244bf4e4c96cbc: Bootstrap starting.
I20260928 23:53:34.286458 24308 tablet_bootstrap.cc:654] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:34.287739 24308 log.cc:824] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:34.287981 24310 tablet_bootstrap.cc:654] T 88049927697445e981ed7096ae0005c1 P fbe8990f78b34e8d8900ec9a243fee74: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:34.288089 24309 tablet_bootstrap.cc:654] T 88049927697445e981ed7096ae0005c1 P 13a24e940bbc4463a0244bf4e4c96cbc: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:34.289250 24309 log.cc:824] T 88049927697445e981ed7096ae0005c1 P 13a24e940bbc4463a0244bf4e4c96cbc: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:34.289449 24310 log.cc:824] T 88049927697445e981ed7096ae0005c1 P fbe8990f78b34e8d8900ec9a243fee74: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:34.289893 24308 tablet_bootstrap.cc:492] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c: No bootstrap required, opened a new log
I20260928 23:53:34.290100 24308 ts_tablet_manager.cc:1397] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:53:34.290957 24309 tablet_bootstrap.cc:492] T 88049927697445e981ed7096ae0005c1 P 13a24e940bbc4463a0244bf4e4c96cbc: No bootstrap required, opened a new log
I20260928 23:53:34.291157 24309 ts_tablet_manager.cc:1397] T 88049927697445e981ed7096ae0005c1 P 13a24e940bbc4463a0244bf4e4c96cbc: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 23:53:34.291349 24310 tablet_bootstrap.cc:492] T 88049927697445e981ed7096ae0005c1 P fbe8990f78b34e8d8900ec9a243fee74: No bootstrap required, opened a new log
I20260928 23:53:34.291524 24310 ts_tablet_manager.cc:1397] T 88049927697445e981ed7096ae0005c1 P fbe8990f78b34e8d8900ec9a243fee74: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:53:34.294021 24309 raft_consensus.cc:348] T 88049927697445e981ed7096ae0005c1 P 13a24e940bbc4463a0244bf4e4c96cbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8d20c28b4446f3a3063914c718ec3c" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42081 } } peers { permanent_uuid: "13a24e940bbc4463a0244bf4e4c96cbc" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46809 } } peers { permanent_uuid: "fbe8990f78b34e8d8900ec9a243fee74" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 32785 } }
I20260928 23:53:34.294030 24308 raft_consensus.cc:348] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8d20c28b4446f3a3063914c718ec3c" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42081 } } peers { permanent_uuid: "13a24e940bbc4463a0244bf4e4c96cbc" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46809 } } peers { permanent_uuid: "fbe8990f78b34e8d8900ec9a243fee74" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 32785 } }
I20260928 23:53:34.294304 24308 raft_consensus.cc:374] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:34.294313 24309 raft_consensus.cc:374] T 88049927697445e981ed7096ae0005c1 P 13a24e940bbc4463a0244bf4e4c96cbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:34.294363 24309 raft_consensus.cc:729] T 88049927697445e981ed7096ae0005c1 P 13a24e940bbc4463a0244bf4e4c96cbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13a24e940bbc4463a0244bf4e4c96cbc, State: Initialized, Role: FOLLOWER
I20260928 23:53:34.294389 24308 raft_consensus.cc:729] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d8d20c28b4446f3a3063914c718ec3c, State: Initialized, Role: FOLLOWER
I20260928 23:53:34.294824 24309 consensus_queue.cc:260] T 88049927697445e981ed7096ae0005c1 P 13a24e940bbc4463a0244bf4e4c96cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8d20c28b4446f3a3063914c718ec3c" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42081 } } peers { permanent_uuid: "13a24e940bbc4463a0244bf4e4c96cbc" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46809 } } peers { permanent_uuid: "fbe8990f78b34e8d8900ec9a243fee74" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 32785 } }
I20260928 23:53:34.294842 24308 consensus_queue.cc:260] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8d20c28b4446f3a3063914c718ec3c" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42081 } } peers { permanent_uuid: "13a24e940bbc4463a0244bf4e4c96cbc" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46809 } } peers { permanent_uuid: "fbe8990f78b34e8d8900ec9a243fee74" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 32785 } }
I20260928 23:53:34.295249 24310 raft_consensus.cc:348] T 88049927697445e981ed7096ae0005c1 P fbe8990f78b34e8d8900ec9a243fee74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8d20c28b4446f3a3063914c718ec3c" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42081 } } peers { permanent_uuid: "13a24e940bbc4463a0244bf4e4c96cbc" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46809 } } peers { permanent_uuid: "fbe8990f78b34e8d8900ec9a243fee74" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 32785 } }
I20260928 23:53:34.295533 24310 raft_consensus.cc:374] T 88049927697445e981ed7096ae0005c1 P fbe8990f78b34e8d8900ec9a243fee74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:34.295625 24310 raft_consensus.cc:729] T 88049927697445e981ed7096ae0005c1 P fbe8990f78b34e8d8900ec9a243fee74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe8990f78b34e8d8900ec9a243fee74, State: Initialized, Role: FOLLOWER
I20260928 23:53:34.295859 24308 ts_tablet_manager.cc:1428] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:53:34.295869 24309 ts_tablet_manager.cc:1428] T 88049927697445e981ed7096ae0005c1 P 13a24e940bbc4463a0244bf4e4c96cbc: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260928 23:53:34.296209 24291 heartbeater.cc:503] Master 127.6.152.62:46481 was elected leader, sending a full tablet report...
I20260928 23:53:34.296355 24023 heartbeater.cc:503] Master 127.6.152.62:46481 was elected leader, sending a full tablet report...
I20260928 23:53:34.296183 24310 consensus_queue.cc:260] T 88049927697445e981ed7096ae0005c1 P fbe8990f78b34e8d8900ec9a243fee74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8d20c28b4446f3a3063914c718ec3c" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42081 } } peers { permanent_uuid: "13a24e940bbc4463a0244bf4e4c96cbc" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46809 } } peers { permanent_uuid: "fbe8990f78b34e8d8900ec9a243fee74" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 32785 } }
I20260928 23:53:34.297780 24310 ts_tablet_manager.cc:1428] T 88049927697445e981ed7096ae0005c1 P fbe8990f78b34e8d8900ec9a243fee74: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:53:34.298103 24157 heartbeater.cc:503] Master 127.6.152.62:46481 was elected leader, sending a full tablet report...
I20260928 23:53:34.322525 24314 raft_consensus.cc:482] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:53:34.322734 24314 raft_consensus.cc:504] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8d20c28b4446f3a3063914c718ec3c" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42081 } } peers { permanent_uuid: "13a24e940bbc4463a0244bf4e4c96cbc" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46809 } } peers { permanent_uuid: "fbe8990f78b34e8d8900ec9a243fee74" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 32785 } }
I20260928 23:53:34.323848 24314 leader_election.cc:288] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13a24e940bbc4463a0244bf4e4c96cbc (127.6.152.3:46809), fbe8990f78b34e8d8900ec9a243fee74 (127.6.152.2:32785)
I20260928 23:53:34.328296 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:34.324237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:43727 (local address 127.6.152.2:32785)
0928 23:53:34.324377 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:34.324381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:34.324653 (+ 272us) server_negotiation.cc:407] Connection header received
0928 23:53:34.324787 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:34.324789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:34.324860 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:34.324939 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:34.325248 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.325717 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.326926 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.327502 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.327602 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:34.327664 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:34.328088 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 23:53:34.328169 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 23:53:34.328317 24307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:34.324182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:33697 (local address 127.6.152.3:46809)
0928 23:53:34.324331 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:34.324336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:34.324519 (+ 183us) server_negotiation.cc:407] Connection header received
0928 23:53:34.324624 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:34.324628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:34.324679 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:34.324755 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:34.325142 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.325678 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.326885 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.327502 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.327602 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:34.327664 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:34.328088 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 23:53:34.328168 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 23:53:34.328346 24318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:34.324105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:32785 (local address 127.6.152.1:43727)
0928 23:53:34.324543 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:34.324559 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:34.324664 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:34.324955 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:34.324957 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:34.324971 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:34.325135 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:34.325142 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.325860 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.325863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:34.326743 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:34.326752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.327725 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.327729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:34.327877 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:34.327973 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 23:53:34.328052 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 23:53:34.328143 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":119,"thread_start_us":129,"threads_started":1}
I20260928 23:53:34.328349 24317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:34.324098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:46809 (local address 127.6.152.1:33697)
0928 23:53:34.324420 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:34.324439 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:34.324535 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:34.324779 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:34.324782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:34.324800 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:34.325029 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:34.325038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.325803 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.325810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:34.326749 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:34.326770 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.327725 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.327729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:34.327935 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:34.327978 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 23:53:34.328052 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 23:53:34.328143 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":168,"thread_start_us":87,"threads_started":1}
I20260928 23:53:34.329264 24245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88049927697445e981ed7096ae0005c1" candidate_uuid: "7d8d20c28b4446f3a3063914c718ec3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13a24e940bbc4463a0244bf4e4c96cbc" is_pre_election: true
I20260928 23:53:34.329259 24111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88049927697445e981ed7096ae0005c1" candidate_uuid: "7d8d20c28b4446f3a3063914c718ec3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe8990f78b34e8d8900ec9a243fee74" is_pre_election: true
I20260928 23:53:34.329587 24111 raft_consensus.cc:2454] T 88049927697445e981ed7096ae0005c1 P fbe8990f78b34e8d8900ec9a243fee74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d8d20c28b4446f3a3063914c718ec3c in term 0.
I20260928 23:53:34.329583 24245 raft_consensus.cc:2454] T 88049927697445e981ed7096ae0005c1 P 13a24e940bbc4463a0244bf4e4c96cbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d8d20c28b4446f3a3063914c718ec3c in term 0.
I20260928 23:53:34.330087 23913 leader_election.cc:302] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 13a24e940bbc4463a0244bf4e4c96cbc, 7d8d20c28b4446f3a3063914c718ec3c, fbe8990f78b34e8d8900ec9a243fee74; no voters:
I20260928 23:53:34.330359 24314 raft_consensus.cc:2781] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:53:34.330457 24314 raft_consensus.cc:482] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:53:34.330526 24314 raft_consensus.cc:3037] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:34.331633 24314 raft_consensus.cc:504] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8d20c28b4446f3a3063914c718ec3c" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42081 } } peers { permanent_uuid: "13a24e940bbc4463a0244bf4e4c96cbc" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46809 } } peers { permanent_uuid: "fbe8990f78b34e8d8900ec9a243fee74" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 32785 } }
I20260928 23:53:34.332050 24314 leader_election.cc:288] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c [CANDIDATE]: Term 1 election: Requested vote from peers 13a24e940bbc4463a0244bf4e4c96cbc (127.6.152.3:46809), fbe8990f78b34e8d8900ec9a243fee74 (127.6.152.2:32785)
I20260928 23:53:34.332392 24245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88049927697445e981ed7096ae0005c1" candidate_uuid: "7d8d20c28b4446f3a3063914c718ec3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13a24e940bbc4463a0244bf4e4c96cbc"
I20260928 23:53:34.332492 24111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88049927697445e981ed7096ae0005c1" candidate_uuid: "7d8d20c28b4446f3a3063914c718ec3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe8990f78b34e8d8900ec9a243fee74"
I20260928 23:53:34.332516 24245 raft_consensus.cc:3037] T 88049927697445e981ed7096ae0005c1 P 13a24e940bbc4463a0244bf4e4c96cbc [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:34.332643 24111 raft_consensus.cc:3037] T 88049927697445e981ed7096ae0005c1 P fbe8990f78b34e8d8900ec9a243fee74 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:34.333674 24245 raft_consensus.cc:2454] T 88049927697445e981ed7096ae0005c1 P 13a24e940bbc4463a0244bf4e4c96cbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d8d20c28b4446f3a3063914c718ec3c in term 1.
I20260928 23:53:34.333729 24111 raft_consensus.cc:2454] T 88049927697445e981ed7096ae0005c1 P fbe8990f78b34e8d8900ec9a243fee74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d8d20c28b4446f3a3063914c718ec3c in term 1.
I20260928 23:53:34.334013 23912 leader_election.cc:302] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13a24e940bbc4463a0244bf4e4c96cbc, 7d8d20c28b4446f3a3063914c718ec3c; no voters:
I20260928 23:53:34.334213 24314 raft_consensus.cc:2781] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:53:34.334406 24314 raft_consensus.cc:686] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c [term 1 LEADER]: Becoming Leader. State: Replica: 7d8d20c28b4446f3a3063914c718ec3c, State: Running, Role: LEADER
I20260928 23:53:34.334759 24314 consensus_queue.cc:237] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8d20c28b4446f3a3063914c718ec3c" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42081 } } peers { permanent_uuid: "13a24e940bbc4463a0244bf4e4c96cbc" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46809 } } peers { permanent_uuid: "fbe8990f78b34e8d8900ec9a243fee74" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 32785 } }
I20260928 23:53:34.337934 23832 catalog_manager.cc:5495] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d8d20c28b4446f3a3063914c718ec3c (127.6.152.1). New cstate: current_term: 1 leader_uuid: "7d8d20c28b4446f3a3063914c718ec3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8d20c28b4446f3a3063914c718ec3c" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 42081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13a24e940bbc4463a0244bf4e4c96cbc" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbe8990f78b34e8d8900ec9a243fee74" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 32785 } health_report { overall_health: UNKNOWN } } }
I20260928 23:53:34.350871 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:34.347419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59522 (local address 127.6.152.1:42081)
0928 23:53:34.347534 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:34.347538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:34.347551 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:53:34.347648 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:34.347650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:34.347698 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:34.347784 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:53:34.348000 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.348524 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.349436 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.349684 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.349844 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:34.350132 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:53:34.350557 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:53:34.350656 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:34.350705 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 23:53:34.350766 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 23:53:34.356964 24245 raft_consensus.cc:1261] T 88049927697445e981ed7096ae0005c1 P 13a24e940bbc4463a0244bf4e4c96cbc [term 1 FOLLOWER]: Refusing update from remote peer 7d8d20c28b4446f3a3063914c718ec3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:53:34.356987 24111 raft_consensus.cc:1261] T 88049927697445e981ed7096ae0005c1 P fbe8990f78b34e8d8900ec9a243fee74 [term 1 FOLLOWER]: Refusing update from remote peer 7d8d20c28b4446f3a3063914c718ec3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:53:34.357738 24319 consensus_queue.cc:1035] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "13a24e940bbc4463a0244bf4e4c96cbc" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:53:34.357963 24314 consensus_queue.cc:1035] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe8990f78b34e8d8900ec9a243fee74" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 32785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:53:34.364966 24322 mvcc.cc:205] Tried to move back new op lower bound from 7334459860398821376 to 7334459860317937664. Current Snapshot: MvccSnapshot[applied={T|T < 7334459860398821376}]
I20260928 23:53:34.367861 24324 mvcc.cc:205] Tried to move back new op lower bound from 7334459860398821376 to 7334459860317937664. Current Snapshot: MvccSnapshot[applied={T|T < 7334459860398821376}]
I20260928 23:53:34.373709 23831 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:53:34.373862 23831 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:53:34.376024 23831 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9e106c46c674e80aa464f6089a3ce45: Sending DeleteTablet for 3 replicas of tablet 88049927697445e981ed7096ae0005c1
I20260928 23:53:34.376708 23957 tablet_service.cc:1514] Processing DeleteTablet for tablet 88049927697445e981ed7096ae0005c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:53:34 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:59510
I20260928 23:53:34.376895 24225 tablet_service.cc:1514] Processing DeleteTablet for tablet 88049927697445e981ed7096ae0005c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:53:34 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:51988
I20260928 23:53:34.376895 24091 tablet_service.cc:1514] Processing DeleteTablet for tablet 88049927697445e981ed7096ae0005c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:53:34 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:56512
I20260928 23:53:34.377213 24332 tablet_replica.cc:330] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c: stopping tablet replica
I20260928 23:53:34.377429 24334 tablet_replica.cc:330] T 88049927697445e981ed7096ae0005c1 P fbe8990f78b34e8d8900ec9a243fee74: stopping tablet replica
I20260928 23:53:34.377712 24334 raft_consensus.cc:2229] T 88049927697445e981ed7096ae0005c1 P fbe8990f78b34e8d8900ec9a243fee74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:53:34.377730 24332 raft_consensus.cc:2229] T 88049927697445e981ed7096ae0005c1 P 7d8d20c28b4446f3a3063914c718ec3c [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:53:34.377959 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 23894
I20260928 23:53:34.378575 24333 tablet_replica.cc:330] T 88049927697445e981ed7096ae0005c1 P 13a24e940bbc4463a0244bf4e4c96cbc: stopping tablet replica
I20260928 23:53:34.378849 24333 raft_consensus.cc:2229] T 88049927697445e981ed7096ae0005c1 P 13a24e940bbc4463a0244bf4e4c96cbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:53:34.379141 24333 raft_consensus.cc:2258] T 88049927697445e981ed7096ae0005c1 P 13a24e940bbc4463a0244bf4e4c96cbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:53:34.379498 24334 raft_consensus.cc:2258] T 88049927697445e981ed7096ae0005c1 P fbe8990f78b34e8d8900ec9a243fee74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:53:34.380990 24333 ts_tablet_manager.cc:1905] T 88049927697445e981ed7096ae0005c1 P 13a24e940bbc4463a0244bf4e4c96cbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:53:34.381150 24334 ts_tablet_manager.cc:1905] T 88049927697445e981ed7096ae0005c1 P fbe8990f78b34e8d8900ec9a243fee74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:53:34.383677 24333 ts_tablet_manager.cc:1918] T 88049927697445e981ed7096ae0005c1 P 13a24e940bbc4463a0244bf4e4c96cbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:53:34.383677 24334 ts_tablet_manager.cc:1918] T 88049927697445e981ed7096ae0005c1 P fbe8990f78b34e8d8900ec9a243fee74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:53:34.383776 24334 log.cc:1196] T 88049927697445e981ed7096ae0005c1 P fbe8990f78b34e8d8900ec9a243fee74: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-1/wal/wals/88049927697445e981ed7096ae0005c1
I20260928 23:53:34.383776 24333 log.cc:1196] T 88049927697445e981ed7096ae0005c1 P 13a24e940bbc4463a0244bf4e4c96cbc: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/88049927697445e981ed7096ae0005c1
I20260928 23:53:34.384119 24333 ts_tablet_manager.cc:1939] T 88049927697445e981ed7096ae0005c1 P 13a24e940bbc4463a0244bf4e4c96cbc: Deleting consensus metadata
I20260928 23:53:34.384130 24334 ts_tablet_manager.cc:1939] T 88049927697445e981ed7096ae0005c1 P fbe8990f78b34e8d8900ec9a243fee74: Deleting consensus metadata
I20260928 23:53:34.385130 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 24028
W20260928 23:53:34.385182 23816 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.152.1:42081 (error 108)
I20260928 23:53:34.385931 23818 catalog_manager.cc:4841] TS 13a24e940bbc4463a0244bf4e4c96cbc (127.6.152.3:46809): tablet 88049927697445e981ed7096ae0005c1 (table test-table [id=ad501de2b89946e9ad97482040ff655b]) successfully deleted
I20260928 23:53:34.389642 24302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:34.389377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:42081 (local address 127.0.0.1:59526)
0928 23:53:34.389505 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:34.389568 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.1:42081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 23:53:34.389891 23816 catalog_manager.cc:4584] TS 7d8d20c28b4446f3a3063914c718ec3c (127.6.152.1:42081): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 88049927697445e981ed7096ae0005c1: Network error: Client connection negotiation failed: client connection to 127.6.152.1:42081: connect: Connection refused (error 111)
I20260928 23:53:34.391230 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 24162
I20260928 23:53:34.391501 24302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:34.391252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:32785 (local address 127.0.0.1:56522)
0928 23:53:34.391369 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:34.391423 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.2:32785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 23:53:34.396752 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 23800
2026-09-28T23:53:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2214 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T23:53:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:53:34Z Disabled control of system clock
I20260928 23:53:34.427044 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:44969
--webserver_interface=127.6.152.62
--webserver_port=0
--builtin_ntp_servers=127.6.152.20:39427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:44969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:53:34.535096 24339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:34.535351 24339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:34.535385 24339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:34.539268 24339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:53:34.539316 24339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:34.539331 24339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:34.539347 24339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:53:34.539362 24339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:53:34.544003 24339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:39427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:44969
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:44969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:34.544966 24339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:34.546231 24339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:34.554312 24345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:34.554361 24347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:34.554384 24344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:34.554486 24339 server_base.cc:1034] running on GCE node
I20260928 23:53:34.555177 24339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:34.556185 24339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:34.557363 24339 hybrid_clock.cc:648] HybridClock initialized: now 1790639614557329 us; error 48 us; skew 500 ppm
I20260928 23:53:34.559310 24339 webserver.cc:460] Webserver started at http://127.6.152.62:43087/ using document root <none> and password file <none>
I20260928 23:53:34.559867 24339 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:34.559960 24339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:34.560185 24339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:34.562274 24339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "0ad2ea37f7004e018c42b3edd19d6ba8"
format_stamp: "Formatted at 2026-09-28 23:53:34 on dist-test-slave-p5bw"
server_key: "b3a350126b441ace380842d846ba68b0"
server_key_iv: "2e06fce79235f14e8870dfcbda5eb887"
server_key_version: "encryptionkey@0"
I20260928 23:53:34.562808 24339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "0ad2ea37f7004e018c42b3edd19d6ba8"
format_stamp: "Formatted at 2026-09-28 23:53:34 on dist-test-slave-p5bw"
server_key: "b3a350126b441ace380842d846ba68b0"
server_key_iv: "2e06fce79235f14e8870dfcbda5eb887"
server_key_version: "encryptionkey@0"
I20260928 23:53:34.566278 24339 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260928 23:53:34.568550 24353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:34.569839 24339 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:53:34.569971 24339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "0ad2ea37f7004e018c42b3edd19d6ba8"
format_stamp: "Formatted at 2026-09-28 23:53:34 on dist-test-slave-p5bw"
server_key: "b3a350126b441ace380842d846ba68b0"
server_key_iv: "2e06fce79235f14e8870dfcbda5eb887"
server_key_version: "encryptionkey@0"
I20260928 23:53:34.570080 24339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:34.582532 24339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:34.583132 24339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:34.583317 24339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:34.590054 24339 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:44969
I20260928 23:53:34.590067 24405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:44969 every 8 connection(s)
I20260928 23:53:34.590934 24339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:53:34.592608 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 24339
I20260928 23:53:34.592712 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:53:34.593011 6752 external_mini_cluster.cc:1428] Setting key 99897a38416e30e4122268f26c90429a
I20260928 23:53:34.594033 24406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:34.599408 24406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ad2ea37f7004e018c42b3edd19d6ba8: Bootstrap starting.
I20260928 23:53:34.601696 24406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ad2ea37f7004e018c42b3edd19d6ba8: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:34.601764 24409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:34.594364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55866 (local address 127.6.152.62:44969)
0928 23:53:34.594955 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:34.594970 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:34.595003 (+ 33us) server_negotiation.cc:407] Connection header received
0928 23:53:34.595617 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:34.595632 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:34.596003 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:34.596320 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:34.596733 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.597735 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.598596 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.599050 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.599209 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:34.600113 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:34.600119 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:34.600130 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:34.600508 (+ 378us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:34.600562 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:34.600581 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:34.600686 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:34.600849 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 23:53:34.601097 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":126,"threads_started":1}
I20260928 23:53:34.602463 24406 log.cc:824] T 00000000000000000000000000000000 P 0ad2ea37f7004e018c42b3edd19d6ba8: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:34.604460 24406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ad2ea37f7004e018c42b3edd19d6ba8: No bootstrap required, opened a new log
I20260928 23:53:34.608000 24406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ad2ea37f7004e018c42b3edd19d6ba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ad2ea37f7004e018c42b3edd19d6ba8" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 44969 } }
I20260928 23:53:34.608235 24406 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ad2ea37f7004e018c42b3edd19d6ba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:34.608312 24406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ad2ea37f7004e018c42b3edd19d6ba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ad2ea37f7004e018c42b3edd19d6ba8, State: Initialized, Role: FOLLOWER
I20260928 23:53:34.608740 24406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ad2ea37f7004e018c42b3edd19d6ba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ad2ea37f7004e018c42b3edd19d6ba8" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 44969 } }
I20260928 23:53:34.608913 24406 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ad2ea37f7004e018c42b3edd19d6ba8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:53:34.608994 24406 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ad2ea37f7004e018c42b3edd19d6ba8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:53:34.609089 24406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ad2ea37f7004e018c42b3edd19d6ba8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:34.609931 24406 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ad2ea37f7004e018c42b3edd19d6ba8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ad2ea37f7004e018c42b3edd19d6ba8" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 44969 } }
I20260928 23:53:34.610248 24406 leader_election.cc:302] T 00000000000000000000000000000000 P 0ad2ea37f7004e018c42b3edd19d6ba8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ad2ea37f7004e018c42b3edd19d6ba8; no voters:
I20260928 23:53:34.610538 24406 leader_election.cc:288] T 00000000000000000000000000000000 P 0ad2ea37f7004e018c42b3edd19d6ba8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:53:34.610658 24411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ad2ea37f7004e018c42b3edd19d6ba8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:53:34.610872 24411 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ad2ea37f7004e018c42b3edd19d6ba8 [term 1 LEADER]: Becoming Leader. State: Replica: 0ad2ea37f7004e018c42b3edd19d6ba8, State: Running, Role: LEADER
I20260928 23:53:34.611146 24411 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ad2ea37f7004e018c42b3edd19d6ba8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ad2ea37f7004e018c42b3edd19d6ba8" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 44969 } }
I20260928 23:53:34.611640 24406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ad2ea37f7004e018c42b3edd19d6ba8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:53:34.612712 24413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ad2ea37f7004e018c42b3edd19d6ba8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ad2ea37f7004e018c42b3edd19d6ba8. Latest consensus state: current_term: 1 leader_uuid: "0ad2ea37f7004e018c42b3edd19d6ba8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ad2ea37f7004e018c42b3edd19d6ba8" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 44969 } } }
I20260928 23:53:34.612879 24413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ad2ea37f7004e018c42b3edd19d6ba8 [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:34.613217 24412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ad2ea37f7004e018c42b3edd19d6ba8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ad2ea37f7004e018c42b3edd19d6ba8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ad2ea37f7004e018c42b3edd19d6ba8" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 44969 } } }
I20260928 23:53:34.613304 24412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ad2ea37f7004e018c42b3edd19d6ba8 [sys.catalog]: This master's current role is: LEADER
W20260928 23:53:34.616068 24427 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ad2ea37f7004e018c42b3edd19d6ba8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:53:34.616153 24427 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:53:34.616226 24420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:53:34.616957 24420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:53:34.621479 24420 catalog_manager.cc:1347] Generated new cluster ID: 1ce93b2f71fd4989a0b3648d80b19abc
I20260928 23:53:34.621542 24420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:53:34.647607 24420 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:53:34.648546 24420 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:53:34.656617 24420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ad2ea37f7004e018c42b3edd19d6ba8: Generated new TSK 0
I20260928 23:53:34.657353 24420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:53:34.664373 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:44969
--builtin_ntp_servers=127.6.152.20:39427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:53:34.774328 24430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:34.774565 24430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:34.774628 24430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:34.778563 24430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:34.778606 24430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:34.778659 24430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:53:34.783286 24430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:39427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:44969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:34.784271 24430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:34.785643 24430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:53:34.795032 24430 server_base.cc:1034] running on GCE node
W20260928 23:53:34.794986 24438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:34.794986 24435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:34.795037 24436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:34.795603 24430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:34.796207 24430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:34.797364 24430 hybrid_clock.cc:648] HybridClock initialized: now 1790639614797343 us; error 39 us; skew 500 ppm
I20260928 23:53:34.799065 24430 webserver.cc:460] Webserver started at http://127.6.152.1:35457/ using document root <none> and password file <none>
I20260928 23:53:34.799616 24430 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:34.799718 24430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:34.799944 24430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:34.801973 24430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "b34b11c39c474bcd9a9a5b04e5a8a4ca"
format_stamp: "Formatted at 2026-09-28 23:53:34 on dist-test-slave-p5bw"
server_key: "c7ff445ed172cfa507f5dfe92a8bd443"
server_key_iv: "26578495cc5b261d243df245763dd811"
server_key_version: "encryptionkey@0"
I20260928 23:53:34.802529 24430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "b34b11c39c474bcd9a9a5b04e5a8a4ca"
format_stamp: "Formatted at 2026-09-28 23:53:34 on dist-test-slave-p5bw"
server_key: "c7ff445ed172cfa507f5dfe92a8bd443"
server_key_iv: "26578495cc5b261d243df245763dd811"
server_key_version: "encryptionkey@0"
I20260928 23:53:34.805984 24430 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:53:34.808254 24444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:34.809417 24430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:53:34.809537 24430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "b34b11c39c474bcd9a9a5b04e5a8a4ca"
format_stamp: "Formatted at 2026-09-28 23:53:34 on dist-test-slave-p5bw"
server_key: "c7ff445ed172cfa507f5dfe92a8bd443"
server_key_iv: "26578495cc5b261d243df245763dd811"
server_key_version: "encryptionkey@0"
I20260928 23:53:34.809638 24430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:34.825336 24430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:34.825922 24430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:34.826097 24430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:34.826617 24430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:53:34.827469 24430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:53:34.827533 24430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:34.827601 24430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:53:34.827641 24430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:34.836272 24430 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:46423
I20260928 23:53:34.836297 24557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:46423 every 8 connection(s)
I20260928 23:53:34.837129 24430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:53:34.840777 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 24430
I20260928 23:53:34.840952 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
I20260928 23:53:34.841305 6752 external_mini_cluster.cc:1428] Setting key edd56e74fb58e58f2ddff5c300a1fe69
I20260928 23:53:34.843299 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:44969
--builtin_ntp_servers=127.6.152.20:39427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 23:53:34.846402 24409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:34.838780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:58805 (local address 127.6.152.62:44969)
0928 23:53:34.838915 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:34.838919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:34.839942 (+ 1023us) server_negotiation.cc:407] Connection header received
0928 23:53:34.841554 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:34.841557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:34.841608 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:34.841695 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:34.842544 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.843324 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.844477 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.844716 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.844853 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:34.845915 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:34.845919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:34.845921 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:34.845934 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:34.845959 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:34.845967 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:34.846037 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:34.846190 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 23:53:34.846264 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:53:34.847275 24560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:34.839157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:44969 (local address 127.6.152.1:58805)
0928 23:53:34.839799 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:34.839838 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:34.841325 (+ 1487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:34.841849 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:34.841856 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:34.841939 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:53:34.842409 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:34.842422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.843464 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.843470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:34.844358 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:34.844367 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:34.844908 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:34.844912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:34.845070 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:34.845662 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:53:34.845664 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:53:34.845703 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:53:34.845769 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:53:34.846050 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:53:34.846061 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:53:34.846071 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 23:53:34.846189 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 23:53:34.846454 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":149,"threads_started":1}
I20260928 23:53:34.848410 24558 heartbeater.cc:347] Connected to a master server at 127.6.152.62:44969
I20260928 23:53:34.848664 24558 heartbeater.cc:464] Registering TS with master...
I20260928 23:53:34.849323 24558 heartbeater.cc:511] Master 127.6.152.62:44969 requested a full tablet report, sending...
I20260928 23:53:34.850813 24370 ts_manager.cc:194] Registered new tserver with Master: b34b11c39c474bcd9a9a5b04e5a8a4ca (127.6.152.1:46423)
I20260928 23:53:34.851961 24370 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.152.1:58805
W20260928 23:53:34.955628 24561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:34.955896 24561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:34.955933 24561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:34.960124 24561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:34.960186 24561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:34.960268 24561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:53:34.965090 24561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:39427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:44969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:34.966169 24561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:34.967553 24561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:34.976888 24567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:34.976888 24569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:34.976894 24566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:34.977156 24561 server_base.cc:1034] running on GCE node
I20260928 23:53:34.977535 24561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:34.978152 24561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:34.979334 24561 hybrid_clock.cc:648] HybridClock initialized: now 1790639614979318 us; error 26 us; skew 500 ppm
I20260928 23:53:34.981348 24561 webserver.cc:460] Webserver started at http://127.6.152.2:45545/ using document root <none> and password file <none>
I20260928 23:53:34.981920 24561 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:34.982009 24561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:34.982232 24561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:34.984266 24561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "5a69eb235abd4e5fa8c485ea25c942cc"
format_stamp: "Formatted at 2026-09-28 23:53:34 on dist-test-slave-p5bw"
server_key: "8538b49a0215084234638c221be94017"
server_key_iv: "d7ff7301c4494f7ba65a9d1fcbfbee35"
server_key_version: "encryptionkey@0"
I20260928 23:53:34.984858 24561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "5a69eb235abd4e5fa8c485ea25c942cc"
format_stamp: "Formatted at 2026-09-28 23:53:34 on dist-test-slave-p5bw"
server_key: "8538b49a0215084234638c221be94017"
server_key_iv: "d7ff7301c4494f7ba65a9d1fcbfbee35"
server_key_version: "encryptionkey@0"
I20260928 23:53:34.988565 24561 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:53:34.990950 24575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:34.992233 24561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 23:53:34.992338 24561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "5a69eb235abd4e5fa8c485ea25c942cc"
format_stamp: "Formatted at 2026-09-28 23:53:34 on dist-test-slave-p5bw"
server_key: "8538b49a0215084234638c221be94017"
server_key_iv: "d7ff7301c4494f7ba65a9d1fcbfbee35"
server_key_version: "encryptionkey@0"
I20260928 23:53:34.992477 24561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:35.017532 24561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:35.018260 24561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:35.018441 24561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:35.019016 24561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:53:35.019964 24561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:53:35.020045 24561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:35.020139 24561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:53:35.020177 24561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:35.028565 24561 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:44717
I20260928 23:53:35.028589 24688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:44717 every 8 connection(s)
I20260928 23:53:35.029479 24561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:53:35.038086 24409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:35.031338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:56027 (local address 127.6.152.62:44969)
0928 23:53:35.031519 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:35.031524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:35.032604 (+ 1080us) server_negotiation.cc:407] Connection header received
0928 23:53:35.033936 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:35.033938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:35.033985 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:35.034058 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:35.034872 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.035377 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.036418 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.036614 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.036686 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:35.037695 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:35.037698 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:35.037700 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:35.037711 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:35.037730 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:35.037738 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:35.037800 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:35.037921 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 23:53:35.037974 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 23:53:35.038915 24691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:35.031792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:44969 (local address 127.6.152.2:56027)
0928 23:53:35.032436 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:35.032476 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:35.033754 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:35.034204 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:35.034210 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:35.034291 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:53:35.034757 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:35.034769 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.035472 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.035475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:35.036324 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:35.036333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.036726 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.036729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:35.036892 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:35.037465 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:53:35.037467 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:53:35.037505 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:53:35.037563 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:53:35.037809 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:53:35.037815 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:53:35.037819 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:53:35.037917 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 23:53:35.038116 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":105,"threads_started":1}
I20260928 23:53:35.039577 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 24561
I20260928 23:53:35.039706 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:53:35.040001 6752 external_mini_cluster.cc:1428] Setting key af129eb0283f22681e49a60831c36a3d
I20260928 23:53:35.040104 24689 heartbeater.cc:347] Connected to a master server at 127.6.152.62:44969
I20260928 23:53:35.040342 24689 heartbeater.cc:464] Registering TS with master...
I20260928 23:53:35.040978 24689 heartbeater.cc:511] Master 127.6.152.62:44969 requested a full tablet report, sending...
I20260928 23:53:35.041862 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:44969
--builtin_ntp_servers=127.6.152.20:39427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 23:53:35.041910 24370 ts_manager.cc:194] Registered new tserver with Master: 5a69eb235abd4e5fa8c485ea25c942cc (127.6.152.2:44717)
I20260928 23:53:35.042459 24370 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.152.2:56027
W20260928 23:53:35.151459 24692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:35.151703 24692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:35.151742 24692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:35.155802 24692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:35.155855 24692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:35.155965 24692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:53:35.161003 24692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:39427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:44969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:35.161971 24692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:35.163281 24692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:35.173111 24700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:35.173134 24697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:35.173398 24692 server_base.cc:1034] running on GCE node
W20260928 23:53:35.173466 24698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:35.173818 24692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:35.174460 24692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:35.175608 24692 hybrid_clock.cc:648] HybridClock initialized: now 1790639615175586 us; error 37 us; skew 500 ppm
I20260928 23:53:35.177367 24692 webserver.cc:460] Webserver started at http://127.6.152.3:41873/ using document root <none> and password file <none>
I20260928 23:53:35.177996 24692 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:35.178089 24692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:35.178315 24692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:35.180531 24692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "8097a7a3512c495aaefffd3a3b5833df"
format_stamp: "Formatted at 2026-09-28 23:53:35 on dist-test-slave-p5bw"
server_key: "0979df35cfda82c0fddeb8fa6b0908ea"
server_key_iv: "c8980532d3afaa7a55ebf8dea2d71530"
server_key_version: "encryptionkey@0"
I20260928 23:53:35.181169 24692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "8097a7a3512c495aaefffd3a3b5833df"
format_stamp: "Formatted at 2026-09-28 23:53:35 on dist-test-slave-p5bw"
server_key: "0979df35cfda82c0fddeb8fa6b0908ea"
server_key_iv: "c8980532d3afaa7a55ebf8dea2d71530"
server_key_version: "encryptionkey@0"
I20260928 23:53:35.185002 24692 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:53:35.187346 24706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:35.188501 24692 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:53:35.188629 24692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "8097a7a3512c495aaefffd3a3b5833df"
format_stamp: "Formatted at 2026-09-28 23:53:35 on dist-test-slave-p5bw"
server_key: "0979df35cfda82c0fddeb8fa6b0908ea"
server_key_iv: "c8980532d3afaa7a55ebf8dea2d71530"
server_key_version: "encryptionkey@0"
I20260928 23:53:35.188730 24692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:35.205955 24692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:35.206650 24692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:35.206844 24692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:35.207427 24692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:53:35.208355 24692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:53:35.208422 24692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:35.208492 24692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:53:35.208532 24692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:35.217451 24692 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:32821
I20260928 23:53:35.217480 24819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:32821 every 8 connection(s)
I20260928 23:53:35.218355 24692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:53:35.227233 24409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:35.220167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:38571 (local address 127.6.152.62:44969)
0928 23:53:35.220320 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:35.220324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:35.221306 (+ 982us) server_negotiation.cc:407] Connection header received
0928 23:53:35.222671 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:35.222675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:35.222722 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:35.222789 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:35.223766 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.224295 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.225471 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.225678 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.225756 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:35.226779 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:35.226781 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:35.226784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:35.226794 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:35.226809 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:35.226815 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:35.226866 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:35.227027 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 23:53:35.227080 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 23:53:35.227955 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 24692
I20260928 23:53:35.228008 24822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:35.220457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:44969 (local address 127.6.152.3:38571)
0928 23:53:35.221147 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:35.221223 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:35.222453 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:35.223011 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:35.223017 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:35.223100 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:53:35.223636 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:35.223650 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.224421 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.224423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:35.225371 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:35.225381 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.225808 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.225811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:35.225947 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:35.226552 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:53:35.226553 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:53:35.226592 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:53:35.226658 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:53:35.226879 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:53:35.226886 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:53:35.226889 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:53:35.226979 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 23:53:35.227193 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":110,"threads_started":1}
I20260928 23:53:35.228086 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:53:35.228422 6752 external_mini_cluster.cc:1428] Setting key 2353f51fe5f0a8ead7f492d0412322c0
I20260928 23:53:35.229549 24820 heartbeater.cc:347] Connected to a master server at 127.6.152.62:44969
I20260928 23:53:35.229791 24820 heartbeater.cc:464] Registering TS with master...
I20260928 23:53:35.230286 24820 heartbeater.cc:511] Master 127.6.152.62:44969 requested a full tablet report, sending...
I20260928 23:53:35.231283 24370 ts_manager.cc:194] Registered new tserver with Master: 8097a7a3512c495aaefffd3a3b5833df (127.6.152.3:32821)
I20260928 23:53:35.231833 24370 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.152.3:38571
I20260928 23:53:35.242117 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:53:35.247694 24409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:35.244030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55878 (local address 127.6.152.62:44969)
0928 23:53:35.244156 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:35.244160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:35.244326 (+ 166us) server_negotiation.cc:407] Connection header received
0928 23:53:35.244478 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:35.244481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:35.244529 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:35.244608 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:35.244921 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.245418 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.246233 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.246453 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.246558 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:35.247254 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:35.247257 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:35.247260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:35.247272 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:35.247296 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:35.247304 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:35.247374 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:35.247506 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 23:53:35.247555 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:53:35.250125 24370 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:53:35.253010 24370 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:53:35.266319 24832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:35.260949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43618 (local address 127.6.152.1:46423)
0928 23:53:35.261445 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:35.261457 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:35.261482 (+ 25us) server_negotiation.cc:407] Connection header received
0928 23:53:35.261953 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:35.261959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:35.262177 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:35.262334 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:35.262557 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.263663 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.264802 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.265608 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.265719 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:35.265897 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:35.266039 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 23:53:35.266144 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":128,"threads_started":1}
I20260928 23:53:35.266346 24831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:35.260676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:46423 (local address 127.0.0.1:43618)
0928 23:53:35.261030 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:35.261085 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:35.261208 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:35.262337 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:35.262341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:35.262361 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:35.262456 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:35.262461 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.263775 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.263779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:35.264679 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:35.264688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.265735 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.265741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:35.265850 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:35.265900 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 23:53:35.266006 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 23:53:35.266182 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":29,"thread_start_us":75,"threads_started":1}
I20260928 23:53:35.266690 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:35.261742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53722 (local address 127.6.152.3:32821)
0928 23:53:35.262125 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:35.262135 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:35.262158 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:53:35.262229 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:35.262238 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:35.262423 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:35.262564 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:35.262768 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.263507 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.264874 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.265608 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.265720 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:35.265897 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:35.266435 (+ 538us) server_negotiation.cc:299] Negotiation successful
0928 23:53:35.266533 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":116,"threads_started":1}
I20260928 23:53:35.266728 24834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:35.261186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:32821 (local address 127.0.0.1:53722)
0928 23:53:35.261578 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:35.261592 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:35.261676 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:35.262553 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:35.262559 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:35.262577 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:35.262654 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:35.262658 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.263626 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.263629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:35.264704 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:35.264712 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.266116 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.266119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:35.266200 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:35.266223 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:53:35.266291 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 23:53:35.266345 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":54,"thread_start_us":112,"threads_started":1}
I20260928 23:53:35.268893 24833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:35.261346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:44717 (local address 127.0.0.1:36128)
0928 23:53:35.262771 (+ 1425us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:35.262784 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:35.262862 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:35.263521 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:35.263524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:35.263548 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:35.263659 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:35.263666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.265788 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.265791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:35.267226 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:35.267236 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.268469 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.268473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:35.268566 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:35.268589 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:53:35.268664 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 23:53:35.268730 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1365,"mutex_wait_us":58,"thread_start_us":108,"threads_started":1}
I20260928 23:53:35.269300 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:35.261592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36128 (local address 127.6.152.2:44717)
0928 23:53:35.263088 (+ 1496us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:35.263098 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:35.263117 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:53:35.263202 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:35.263207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:35.263378 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:35.263514 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:35.264850 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.265649 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.267364 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.268138 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.268237 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:35.268406 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:35.269065 (+ 659us) server_negotiation.cc:299] Negotiation successful
0928 23:53:35.269155 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1399,"thread_start_us":117,"threads_started":1}
I20260928 23:53:35.269560 24754 tablet_service.cc:1467] Processing CreateTablet for tablet ab9996746fe542b4aaf931164a11177d (DEFAULT_TABLE table=test-table [id=4e5344282d884b3987151e14a3fd4e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:53:35.269574 24492 tablet_service.cc:1467] Processing CreateTablet for tablet ab9996746fe542b4aaf931164a11177d (DEFAULT_TABLE table=test-table [id=4e5344282d884b3987151e14a3fd4e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:53:35.270484 24754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9996746fe542b4aaf931164a11177d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:35.270485 24492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9996746fe542b4aaf931164a11177d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:35.275753 24623 tablet_service.cc:1467] Processing CreateTablet for tablet ab9996746fe542b4aaf931164a11177d (DEFAULT_TABLE table=test-table [id=4e5344282d884b3987151e14a3fd4e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:53:35.277069 24837 tablet_bootstrap.cc:492] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df: Bootstrap starting.
I20260928 23:53:35.277416 24623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9996746fe542b4aaf931164a11177d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:35.277688 24838 tablet_bootstrap.cc:492] T ab9996746fe542b4aaf931164a11177d P b34b11c39c474bcd9a9a5b04e5a8a4ca: Bootstrap starting.
I20260928 23:53:35.278996 24837 tablet_bootstrap.cc:654] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:35.279687 24838 tablet_bootstrap.cc:654] T ab9996746fe542b4aaf931164a11177d P b34b11c39c474bcd9a9a5b04e5a8a4ca: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:35.279685 24837 log.cc:824] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:35.280501 24838 log.cc:824] T ab9996746fe542b4aaf931164a11177d P b34b11c39c474bcd9a9a5b04e5a8a4ca: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:35.281587 24837 tablet_bootstrap.cc:492] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df: No bootstrap required, opened a new log
I20260928 23:53:35.281816 24837 ts_tablet_manager.cc:1397] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 23:53:35.282389 24838 tablet_bootstrap.cc:492] T ab9996746fe542b4aaf931164a11177d P b34b11c39c474bcd9a9a5b04e5a8a4ca: No bootstrap required, opened a new log
I20260928 23:53:35.282558 24838 ts_tablet_manager.cc:1397] T ab9996746fe542b4aaf931164a11177d P b34b11c39c474bcd9a9a5b04e5a8a4ca: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 23:53:35.284572 24841 tablet_bootstrap.cc:492] T ab9996746fe542b4aaf931164a11177d P 5a69eb235abd4e5fa8c485ea25c942cc: Bootstrap starting.
I20260928 23:53:35.285358 24837 raft_consensus.cc:348] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34b11c39c474bcd9a9a5b04e5a8a4ca" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 46423 } } peers { permanent_uuid: "5a69eb235abd4e5fa8c485ea25c942cc" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 44717 } } peers { permanent_uuid: "8097a7a3512c495aaefffd3a3b5833df" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 32821 } }
I20260928 23:53:35.285640 24837 raft_consensus.cc:374] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:35.285694 24837 raft_consensus.cc:729] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8097a7a3512c495aaefffd3a3b5833df, State: Initialized, Role: FOLLOWER
I20260928 23:53:35.286120 24837 consensus_queue.cc:260] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34b11c39c474bcd9a9a5b04e5a8a4ca" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 46423 } } peers { permanent_uuid: "5a69eb235abd4e5fa8c485ea25c942cc" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 44717 } } peers { permanent_uuid: "8097a7a3512c495aaefffd3a3b5833df" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 32821 } }
I20260928 23:53:35.286463 24838 raft_consensus.cc:348] T ab9996746fe542b4aaf931164a11177d P b34b11c39c474bcd9a9a5b04e5a8a4ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34b11c39c474bcd9a9a5b04e5a8a4ca" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 46423 } } peers { permanent_uuid: "5a69eb235abd4e5fa8c485ea25c942cc" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 44717 } } peers { permanent_uuid: "8097a7a3512c495aaefffd3a3b5833df" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 32821 } }
I20260928 23:53:35.286703 24838 raft_consensus.cc:374] T ab9996746fe542b4aaf931164a11177d P b34b11c39c474bcd9a9a5b04e5a8a4ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:35.286746 24841 tablet_bootstrap.cc:654] T ab9996746fe542b4aaf931164a11177d P 5a69eb235abd4e5fa8c485ea25c942cc: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:35.286753 24838 raft_consensus.cc:729] T ab9996746fe542b4aaf931164a11177d P b34b11c39c474bcd9a9a5b04e5a8a4ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b34b11c39c474bcd9a9a5b04e5a8a4ca, State: Initialized, Role: FOLLOWER
I20260928 23:53:35.286806 24820 heartbeater.cc:503] Master 127.6.152.62:44969 was elected leader, sending a full tablet report...
I20260928 23:53:35.287101 24837 ts_tablet_manager.cc:1428] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 23:53:35.287163 24838 consensus_queue.cc:260] T ab9996746fe542b4aaf931164a11177d P b34b11c39c474bcd9a9a5b04e5a8a4ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34b11c39c474bcd9a9a5b04e5a8a4ca" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 46423 } } peers { permanent_uuid: "5a69eb235abd4e5fa8c485ea25c942cc" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 44717 } } peers { permanent_uuid: "8097a7a3512c495aaefffd3a3b5833df" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 32821 } }
I20260928 23:53:35.287514 24841 log.cc:824] T ab9996746fe542b4aaf931164a11177d P 5a69eb235abd4e5fa8c485ea25c942cc: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:35.287770 24558 heartbeater.cc:503] Master 127.6.152.62:44969 was elected leader, sending a full tablet report...
I20260928 23:53:35.289089 24838 ts_tablet_manager.cc:1428] T ab9996746fe542b4aaf931164a11177d P b34b11c39c474bcd9a9a5b04e5a8a4ca: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260928 23:53:35.289593 24841 tablet_bootstrap.cc:492] T ab9996746fe542b4aaf931164a11177d P 5a69eb235abd4e5fa8c485ea25c942cc: No bootstrap required, opened a new log
I20260928 23:53:35.289809 24841 ts_tablet_manager.cc:1397] T ab9996746fe542b4aaf931164a11177d P 5a69eb235abd4e5fa8c485ea25c942cc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 23:53:35.293469 24841 raft_consensus.cc:348] T ab9996746fe542b4aaf931164a11177d P 5a69eb235abd4e5fa8c485ea25c942cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34b11c39c474bcd9a9a5b04e5a8a4ca" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 46423 } } peers { permanent_uuid: "5a69eb235abd4e5fa8c485ea25c942cc" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 44717 } } peers { permanent_uuid: "8097a7a3512c495aaefffd3a3b5833df" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 32821 } }
I20260928 23:53:35.293675 24841 raft_consensus.cc:374] T ab9996746fe542b4aaf931164a11177d P 5a69eb235abd4e5fa8c485ea25c942cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:35.293758 24841 raft_consensus.cc:729] T ab9996746fe542b4aaf931164a11177d P 5a69eb235abd4e5fa8c485ea25c942cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a69eb235abd4e5fa8c485ea25c942cc, State: Initialized, Role: FOLLOWER
I20260928 23:53:35.294103 24841 consensus_queue.cc:260] T ab9996746fe542b4aaf931164a11177d P 5a69eb235abd4e5fa8c485ea25c942cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34b11c39c474bcd9a9a5b04e5a8a4ca" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 46423 } } peers { permanent_uuid: "5a69eb235abd4e5fa8c485ea25c942cc" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 44717 } } peers { permanent_uuid: "8097a7a3512c495aaefffd3a3b5833df" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 32821 } }
I20260928 23:53:35.294659 24689 heartbeater.cc:503] Master 127.6.152.62:44969 was elected leader, sending a full tablet report...
I20260928 23:53:35.294946 24841 ts_tablet_manager.cc:1428] T ab9996746fe542b4aaf931164a11177d P 5a69eb235abd4e5fa8c485ea25c942cc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260928 23:53:35.338225 24559 tablet.cc:2369] T ab9996746fe542b4aaf931164a11177d P b34b11c39c474bcd9a9a5b04e5a8a4ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:53:35.419786 24842 raft_consensus.cc:482] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:53:35.419970 24842 raft_consensus.cc:504] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34b11c39c474bcd9a9a5b04e5a8a4ca" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 46423 } } peers { permanent_uuid: "5a69eb235abd4e5fa8c485ea25c942cc" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 44717 } } peers { permanent_uuid: "8097a7a3512c495aaefffd3a3b5833df" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 32821 } }
I20260928 23:53:35.421098 24842 leader_election.cc:288] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b34b11c39c474bcd9a9a5b04e5a8a4ca (127.6.152.1:46423), 5a69eb235abd4e5fa8c485ea25c942cc (127.6.152.2:44717)
I20260928 23:53:35.425662 24832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:35.421367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:44383 (local address 127.6.152.1:46423)
0928 23:53:35.421519 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:35.421524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:35.421540 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:53:35.421621 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:35.421624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:35.421678 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:35.421739 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:35.422060 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.422819 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.424400 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.425017 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.425123 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:35.425180 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:35.425470 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 23:53:35.425536 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 23:53:35.425662 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:35.421538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:42831 (local address 127.6.152.2:44717)
0928 23:53:35.421657 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:35.421662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:35.421878 (+ 216us) server_negotiation.cc:407] Connection header received
0928 23:53:35.421968 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:35.421970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:35.422038 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:35.422108 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:35.422354 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.423025 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.424406 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.425028 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.425123 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:35.425180 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:35.425470 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 23:53:35.425536 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:53:35.425719 24822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:35.421271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:46423 (local address 127.6.152.3:44383)
0928 23:53:35.421415 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:35.421430 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:35.421524 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:35.421784 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:35.421788 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:35.421812 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:35.421944 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:35.421954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.422925 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.422927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:35.424261 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:35.424274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.425131 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.425135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:35.425322 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:35.425368 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 23:53:35.425445 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 23:53:35.425558 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":159}
I20260928 23:53:35.425913 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:35.421340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:44717 (local address 127.6.152.3:42831)
0928 23:53:35.421784 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:35.421796 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:35.421884 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:35.422122 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:35.422124 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:35.422139 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:35.422228 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:35.422232 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.423111 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.423114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:35.424303 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:35.424313 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.425131 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.425135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:35.425327 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:35.425364 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 23:53:35.425445 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 23:53:35.425558 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":79,"thread_start_us":128,"threads_started":1}
I20260928 23:53:35.426388 24512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9996746fe542b4aaf931164a11177d" candidate_uuid: "8097a7a3512c495aaefffd3a3b5833df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b34b11c39c474bcd9a9a5b04e5a8a4ca" is_pre_election: true
I20260928 23:53:35.426687 24512 raft_consensus.cc:2454] T ab9996746fe542b4aaf931164a11177d P b34b11c39c474bcd9a9a5b04e5a8a4ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8097a7a3512c495aaefffd3a3b5833df in term 0.
I20260928 23:53:35.426684 24643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9996746fe542b4aaf931164a11177d" candidate_uuid: "8097a7a3512c495aaefffd3a3b5833df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a69eb235abd4e5fa8c485ea25c942cc" is_pre_election: true
I20260928 23:53:35.426966 24643 raft_consensus.cc:2454] T ab9996746fe542b4aaf931164a11177d P 5a69eb235abd4e5fa8c485ea25c942cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8097a7a3512c495aaefffd3a3b5833df in term 0.
I20260928 23:53:35.427169 24708 leader_election.cc:302] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8097a7a3512c495aaefffd3a3b5833df, b34b11c39c474bcd9a9a5b04e5a8a4ca; no voters:
I20260928 23:53:35.427438 24842 raft_consensus.cc:2781] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:53:35.427522 24842 raft_consensus.cc:482] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:53:35.427564 24842 raft_consensus.cc:3037] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:35.428541 24842 raft_consensus.cc:504] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34b11c39c474bcd9a9a5b04e5a8a4ca" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 46423 } } peers { permanent_uuid: "5a69eb235abd4e5fa8c485ea25c942cc" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 44717 } } peers { permanent_uuid: "8097a7a3512c495aaefffd3a3b5833df" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 32821 } }
I20260928 23:53:35.428900 24842 leader_election.cc:288] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df [CANDIDATE]: Term 1 election: Requested vote from peers b34b11c39c474bcd9a9a5b04e5a8a4ca (127.6.152.1:46423), 5a69eb235abd4e5fa8c485ea25c942cc (127.6.152.2:44717)
I20260928 23:53:35.429194 24512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9996746fe542b4aaf931164a11177d" candidate_uuid: "8097a7a3512c495aaefffd3a3b5833df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b34b11c39c474bcd9a9a5b04e5a8a4ca"
I20260928 23:53:35.429260 24643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9996746fe542b4aaf931164a11177d" candidate_uuid: "8097a7a3512c495aaefffd3a3b5833df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a69eb235abd4e5fa8c485ea25c942cc"
I20260928 23:53:35.429358 24512 raft_consensus.cc:3037] T ab9996746fe542b4aaf931164a11177d P b34b11c39c474bcd9a9a5b04e5a8a4ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:35.429373 24643 raft_consensus.cc:3037] T ab9996746fe542b4aaf931164a11177d P 5a69eb235abd4e5fa8c485ea25c942cc [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:35.430344 24512 raft_consensus.cc:2454] T ab9996746fe542b4aaf931164a11177d P b34b11c39c474bcd9a9a5b04e5a8a4ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8097a7a3512c495aaefffd3a3b5833df in term 1.
I20260928 23:53:35.430423 24643 raft_consensus.cc:2454] T ab9996746fe542b4aaf931164a11177d P 5a69eb235abd4e5fa8c485ea25c942cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8097a7a3512c495aaefffd3a3b5833df in term 1.
I20260928 23:53:35.430648 24708 leader_election.cc:302] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8097a7a3512c495aaefffd3a3b5833df, b34b11c39c474bcd9a9a5b04e5a8a4ca; no voters:
I20260928 23:53:35.430847 24842 raft_consensus.cc:2781] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:53:35.431047 24842 raft_consensus.cc:686] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df [term 1 LEADER]: Becoming Leader. State: Replica: 8097a7a3512c495aaefffd3a3b5833df, State: Running, Role: LEADER
I20260928 23:53:35.431375 24842 consensus_queue.cc:237] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34b11c39c474bcd9a9a5b04e5a8a4ca" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 46423 } } peers { permanent_uuid: "5a69eb235abd4e5fa8c485ea25c942cc" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 44717 } } peers { permanent_uuid: "8097a7a3512c495aaefffd3a3b5833df" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 32821 } }
I20260928 23:53:35.434516 24370 catalog_manager.cc:5495] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df reported cstate change: term changed from 0 to 1, leader changed from <none> to 8097a7a3512c495aaefffd3a3b5833df (127.6.152.3). New cstate: current_term: 1 leader_uuid: "8097a7a3512c495aaefffd3a3b5833df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b34b11c39c474bcd9a9a5b04e5a8a4ca" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 46423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a69eb235abd4e5fa8c485ea25c942cc" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 44717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8097a7a3512c495aaefffd3a3b5833df" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 32821 } health_report { overall_health: HEALTHY } } }
I20260928 23:53:35.453064 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:35.449908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53726 (local address 127.6.152.3:32821)
0928 23:53:35.450031 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:35.450034 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:35.450050 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:53:35.450199 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:35.450201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:35.450252 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:35.450346 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:53:35.450548 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.451030 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.451801 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:35.452007 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:35.452083 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:35.452314 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:53:35.452707 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:53:35.452796 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:35.452902 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 23:53:35.452947 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 23:53:35.458550 24643 raft_consensus.cc:1261] T ab9996746fe542b4aaf931164a11177d P 5a69eb235abd4e5fa8c485ea25c942cc [term 1 FOLLOWER]: Refusing update from remote peer 8097a7a3512c495aaefffd3a3b5833df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:53:35.458550 24512 raft_consensus.cc:1261] T ab9996746fe542b4aaf931164a11177d P b34b11c39c474bcd9a9a5b04e5a8a4ca [term 1 FOLLOWER]: Refusing update from remote peer 8097a7a3512c495aaefffd3a3b5833df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:53:35.459199 24847 consensus_queue.cc:1035] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df [LEADER]: Connected to new peer: Peer: permanent_uuid: "b34b11c39c474bcd9a9a5b04e5a8a4ca" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 46423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:53:35.459375 24842 consensus_queue.cc:1035] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a69eb235abd4e5fa8c485ea25c942cc" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 44717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:53:35.466318 24850 mvcc.cc:205] Tried to move back new op lower bound from 7334459864911552512 to 7334459864809467904. Current Snapshot: MvccSnapshot[applied={T|T < 7334459864911552512}]
I20260928 23:53:35.467106 24852 mvcc.cc:205] Tried to move back new op lower bound from 7334459864911552512 to 7334459864809467904. Current Snapshot: MvccSnapshot[applied={T|T < 7334459864911552512}]
I20260928 23:53:35.467301 24851 mvcc.cc:205] Tried to move back new op lower bound from 7334459864911552512 to 7334459864809467904. Current Snapshot: MvccSnapshot[applied={T|T < 7334459864911552512}]
I20260928 23:53:35.475083 24370 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:53:35.475250 24370 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:53:35.477424 24370 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ad2ea37f7004e018c42b3edd19d6ba8: Sending DeleteTablet for 3 replicas of tablet ab9996746fe542b4aaf931164a11177d
I20260928 23:53:35.478116 24492 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9996746fe542b4aaf931164a11177d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:53:35 UTC) from {username='slave'} at 127.0.0.1:43618
I20260928 23:53:35.478137 24623 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9996746fe542b4aaf931164a11177d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:53:35 UTC) from {username='slave'} at 127.0.0.1:36128
I20260928 23:53:35.478560 24860 tablet_replica.cc:330] T ab9996746fe542b4aaf931164a11177d P 5a69eb235abd4e5fa8c485ea25c942cc: stopping tablet replica
I20260928 23:53:35.478878 24754 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9996746fe542b4aaf931164a11177d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:53:35 UTC) from {username='slave'} at 127.0.0.1:53722
I20260928 23:53:35.478946 24860 raft_consensus.cc:2229] T ab9996746fe542b4aaf931164a11177d P 5a69eb235abd4e5fa8c485ea25c942cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:53:35.479327 24861 tablet_replica.cc:330] T ab9996746fe542b4aaf931164a11177d P b34b11c39c474bcd9a9a5b04e5a8a4ca: stopping tablet replica
I20260928 23:53:35.479432 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 24430
I20260928 23:53:35.479607 24861 raft_consensus.cc:2229] T ab9996746fe542b4aaf931164a11177d P b34b11c39c474bcd9a9a5b04e5a8a4ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:53:35.480127 24860 raft_consensus.cc:2258] T ab9996746fe542b4aaf931164a11177d P 5a69eb235abd4e5fa8c485ea25c942cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:53:35.481771 24860 ts_tablet_manager.cc:1905] T ab9996746fe542b4aaf931164a11177d P 5a69eb235abd4e5fa8c485ea25c942cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:53:35.485025 24862 tablet_replica.cc:330] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df: stopping tablet replica
I20260928 23:53:35.485350 24862 raft_consensus.cc:2229] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:53:35.485384 24860 ts_tablet_manager.cc:1918] T ab9996746fe542b4aaf931164a11177d P 5a69eb235abd4e5fa8c485ea25c942cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:53:35.485505 24860 log.cc:1196] T ab9996746fe542b4aaf931164a11177d P 5a69eb235abd4e5fa8c485ea25c942cc: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-1/wal/wals/ab9996746fe542b4aaf931164a11177d
I20260928 23:53:35.485850 24860 ts_tablet_manager.cc:1939] T ab9996746fe542b4aaf931164a11177d P 5a69eb235abd4e5fa8c485ea25c942cc: Deleting consensus metadata
I20260928 23:53:35.485882 24862 raft_consensus.cc:2258] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:53:35.487161 24354 catalog_manager.cc:4841] TS 5a69eb235abd4e5fa8c485ea25c942cc (127.6.152.2:44717): tablet ab9996746fe542b4aaf931164a11177d (table test-table [id=4e5344282d884b3987151e14a3fd4e4b]) successfully deleted
I20260928 23:53:35.487615 24862 ts_tablet_manager.cc:1905] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 23:53:35.488122 24355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.152.1:46423 (error 108)
I20260928 23:53:35.489159 24833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:35.488866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:46423 (local address 127.0.0.1:43622)
0928 23:53:35.488978 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:35.489055 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.1:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 23:53:35.489255 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 24561
W20260928 23:53:35.489836 24355 catalog_manager.cc:4584] TS b34b11c39c474bcd9a9a5b04e5a8a4ca (127.6.152.1:46423): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ab9996746fe542b4aaf931164a11177d: Network error: Client connection negotiation failed: client connection to 127.6.152.1:46423: connect: Connection refused (error 111)
I20260928 23:53:35.490212 24862 ts_tablet_manager.cc:1918] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:53:35.490356 24862 log.cc:1196] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/ab9996746fe542b4aaf931164a11177d
I20260928 23:53:35.490653 24862 ts_tablet_manager.cc:1939] T ab9996746fe542b4aaf931164a11177d P 8097a7a3512c495aaefffd3a3b5833df: Deleting consensus metadata
I20260928 23:53:35.491876 24356 catalog_manager.cc:4841] TS 8097a7a3512c495aaefffd3a3b5833df (127.6.152.3:32821): tablet ab9996746fe542b4aaf931164a11177d (table test-table [id=4e5344282d884b3987151e14a3fd4e4b]) successfully deleted
I20260928 23:53:35.496956 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 24692
I20260928 23:53:35.501808 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 24339
2026-09-28T23:53:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1104 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:53:35 dist-test-slave-p5bw krb5kdc[24866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:53:35 dist-test-slave-p5bw krb5kdc[24866](info): set up 2 sockets
Sep 28 23:53:35 dist-test-slave-p5bw krb5kdc[24866](info): commencing operation
krb5kdc: starting...
W20260928 23:53:36.541359 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:53:36 dist-test-slave-p5bw krb5kdc[24866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:53:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:53:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:53:36.660274 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:40121
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:43143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:40121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:53:36.767336 24879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:36.767617 24879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:36.767684 24879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:36.771590 24879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:53:36.771675 24879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:36.771715 24879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:36.771775 24879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:53:36.771812 24879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:53:36.776932 24879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:40121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:40121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:36.777956 24879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:36.778950 24879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:36.788226 24887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:36.788282 24884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:36.788568 24885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:36.788609 24879 server_base.cc:1034] running on GCE node
I20260928 23:53:36.789318 24879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:36.790853 24879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:36.792042 24879 hybrid_clock.cc:648] HybridClock initialized: now 1790639616792020 us; error 44 us; skew 500 ppm
Sep 28 23:53:36 dist-test-slave-p5bw krb5kdc[24866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639616, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:36.795086 24879 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:53:36.795990 24879 webserver.cc:460] Webserver started at http://127.6.152.62:38017/ using document root <none> and password file <none>
I20260928 23:53:36.796540 24879 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:36.796624 24879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:36.796870 24879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:36.798851 24879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "c819fa23773746daa90449cc5c82b09e"
format_stamp: "Formatted at 2026-09-28 23:53:36 on dist-test-slave-p5bw"
server_key: "30ef76015e65d920c2aec8aa86892a1f"
server_key_iv: "949bf7b1d14e2e72030ff05d92ebab31"
server_key_version: "encryptionkey@0"
I20260928 23:53:36.799400 24879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "c819fa23773746daa90449cc5c82b09e"
format_stamp: "Formatted at 2026-09-28 23:53:36 on dist-test-slave-p5bw"
server_key: "30ef76015e65d920c2aec8aa86892a1f"
server_key_iv: "949bf7b1d14e2e72030ff05d92ebab31"
server_key_version: "encryptionkey@0"
I20260928 23:53:36.802935 24879 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260928 23:53:36.805335 24894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:36.806437 24879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:53:36.806563 24879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "c819fa23773746daa90449cc5c82b09e"
format_stamp: "Formatted at 2026-09-28 23:53:36 on dist-test-slave-p5bw"
server_key: "30ef76015e65d920c2aec8aa86892a1f"
server_key_iv: "949bf7b1d14e2e72030ff05d92ebab31"
server_key_version: "encryptionkey@0"
I20260928 23:53:36.806663 24879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:36.817770 24879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:36.820735 24879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:36.820951 24879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:36.827433 24879 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:40121
I20260928 23:53:36.827456 24946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:40121 every 8 connection(s)
I20260928 23:53:36.828302 24879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:53:36.830843 24947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:36.836042 24947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c819fa23773746daa90449cc5c82b09e: Bootstrap starting.
I20260928 23:53:36.836133 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 24879
I20260928 23:53:36.836263 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:53:36.836529 6752 external_mini_cluster.cc:1428] Setting key 1ac55c2b744ff30ae884e280aca30035
I20260928 23:53:36.838652 24947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c819fa23773746daa90449cc5c82b09e: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:36.839485 24947 log.cc:824] T 00000000000000000000000000000000 P c819fa23773746daa90449cc5c82b09e: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:36.841555 24947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c819fa23773746daa90449cc5c82b09e: No bootstrap required, opened a new log
I20260928 23:53:36.845330 24947 raft_consensus.cc:348] T 00000000000000000000000000000000 P c819fa23773746daa90449cc5c82b09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c819fa23773746daa90449cc5c82b09e" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40121 } }
I20260928 23:53:36.845588 24947 raft_consensus.cc:374] T 00000000000000000000000000000000 P c819fa23773746daa90449cc5c82b09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:36.845650 24947 raft_consensus.cc:729] T 00000000000000000000000000000000 P c819fa23773746daa90449cc5c82b09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c819fa23773746daa90449cc5c82b09e, State: Initialized, Role: FOLLOWER
Sep 28 23:53:36 dist-test-slave-p5bw krb5kdc[24866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:53:36.846118 24947 consensus_queue.cc:260] T 00000000000000000000000000000000 P c819fa23773746daa90449cc5c82b09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c819fa23773746daa90449cc5c82b09e" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40121 } }
I20260928 23:53:36.846280 24947 raft_consensus.cc:388] T 00000000000000000000000000000000 P c819fa23773746daa90449cc5c82b09e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:53:36.846357 24947 raft_consensus.cc:482] T 00000000000000000000000000000000 P c819fa23773746daa90449cc5c82b09e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:53:36.846474 24947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c819fa23773746daa90449cc5c82b09e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:36.847307 24947 raft_consensus.cc:504] T 00000000000000000000000000000000 P c819fa23773746daa90449cc5c82b09e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c819fa23773746daa90449cc5c82b09e" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40121 } }
I20260928 23:53:36.847627 24947 leader_election.cc:302] T 00000000000000000000000000000000 P c819fa23773746daa90449cc5c82b09e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c819fa23773746daa90449cc5c82b09e; no voters:
I20260928 23:53:36.847945 24947 leader_election.cc:288] T 00000000000000000000000000000000 P c819fa23773746daa90449cc5c82b09e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:53:36.848016 24952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c819fa23773746daa90449cc5c82b09e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:53:36.848217 24952 raft_consensus.cc:686] T 00000000000000000000000000000000 P c819fa23773746daa90449cc5c82b09e [term 1 LEADER]: Becoming Leader. State: Replica: c819fa23773746daa90449cc5c82b09e, State: Running, Role: LEADER
I20260928 23:53:36.848546 24952 consensus_queue.cc:237] T 00000000000000000000000000000000 P c819fa23773746daa90449cc5c82b09e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c819fa23773746daa90449cc5c82b09e" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40121 } }
I20260928 23:53:36.849186 24947 sys_catalog.cc:564] T 00000000000000000000000000000000 P c819fa23773746daa90449cc5c82b09e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:53:36.849720 24952 sys_catalog.cc:455] T 00000000000000000000000000000000 P c819fa23773746daa90449cc5c82b09e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c819fa23773746daa90449cc5c82b09e. Latest consensus state: current_term: 1 leader_uuid: "c819fa23773746daa90449cc5c82b09e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c819fa23773746daa90449cc5c82b09e" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40121 } } }
I20260928 23:53:36.849942 24952 sys_catalog.cc:458] T 00000000000000000000000000000000 P c819fa23773746daa90449cc5c82b09e [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:36.850134 24953 sys_catalog.cc:455] T 00000000000000000000000000000000 P c819fa23773746daa90449cc5c82b09e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c819fa23773746daa90449cc5c82b09e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c819fa23773746daa90449cc5c82b09e" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40121 } } }
I20260928 23:53:36.850284 24953 sys_catalog.cc:458] T 00000000000000000000000000000000 P c819fa23773746daa90449cc5c82b09e [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:36.850415 24961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:53:36.851995 24950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:36.837902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48192 (local address 127.6.152.62:40121)
0928 23:53:36.838306 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:36.838324 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:36.838365 (+ 41us) server_negotiation.cc:407] Connection header received
0928 23:53:36.839199 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:36.839220 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:36.839625 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:36.839969 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:36.840759 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:36.842200 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:36.843121 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:36.843811 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:36.844007 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:36.846712 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:36.846718 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:36.846730 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:53:36.846756 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:36.848767 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:36.849236 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:36.849242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:36.849250 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:36.849411 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:36.849749 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:36.849753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:36.849754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:36.850303 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:36.850564 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:36.850845 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 23:53:36.851157 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":104,"threads_started":1}
I20260928 23:53:36.853703 24961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:53:36.859000 24961 catalog_manager.cc:1347] Generated new cluster ID: 1f1a6009fea7467abcd96a3af47ede4a
I20260928 23:53:36.859062 24961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:53:36.885360 24961 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:53:36.886307 24961 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:53:36.894598 24961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c819fa23773746daa90449cc5c82b09e: Generated new TSK 0
I20260928 23:53:36.895282 24961 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
I20260928 23:53:36.929365 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:40121
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:43143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:53:37.070729 24973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:37.070976 24973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:37.071017 24973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:37.074869 24973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:37.074915 24973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:37.074970 24973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:53:37.079986 24973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:40121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:37.080924 24973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:37.081847 24973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:37.090788 24979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:37.090797 24981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:37.090902 24973 server_base.cc:1034] running on GCE node
W20260928 23:53:37.090804 24978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:37.091389 24973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:37.092000 24973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:37.093168 24973 hybrid_clock.cc:648] HybridClock initialized: now 1790639617093137 us; error 42 us; skew 500 ppm
Sep 28 23:53:37 dist-test-slave-p5bw krb5kdc[24866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639617, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:37.095680 24973 init.cc:377] Logged in from keytab as kudu/127.6.152.1@KRBTEST.COM (short username kudu)
I20260928 23:53:37.096499 24973 webserver.cc:460] Webserver started at http://127.6.152.1:44779/ using document root <none> and password file <none>
I20260928 23:53:37.097086 24973 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:37.097172 24973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:37.097384 24973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:37.099277 24973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "a60a591f2cad4b2aab975492b6b55b47"
format_stamp: "Formatted at 2026-09-28 23:53:37 on dist-test-slave-p5bw"
server_key: "e98cdc94139b795fe11884d9a7f5496d"
server_key_iv: "b5a8324a226da00d529f5321f79bb305"
server_key_version: "encryptionkey@0"
I20260928 23:53:37.099812 24973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "a60a591f2cad4b2aab975492b6b55b47"
format_stamp: "Formatted at 2026-09-28 23:53:37 on dist-test-slave-p5bw"
server_key: "e98cdc94139b795fe11884d9a7f5496d"
server_key_iv: "b5a8324a226da00d529f5321f79bb305"
server_key_version: "encryptionkey@0"
I20260928 23:53:37.103266 24973 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 23:53:37.105549 24988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:37.106681 24973 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:53:37.106804 24973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "a60a591f2cad4b2aab975492b6b55b47"
format_stamp: "Formatted at 2026-09-28 23:53:37 on dist-test-slave-p5bw"
server_key: "e98cdc94139b795fe11884d9a7f5496d"
server_key_iv: "b5a8324a226da00d529f5321f79bb305"
server_key_version: "encryptionkey@0"
I20260928 23:53:37.106912 24973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:37.119434 24973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:37.122139 24973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:37.122328 24973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:37.122870 24973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:53:37.123653 24973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:53:37.123719 24973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:37.123786 24973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:53:37.123832 24973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:37.132412 24973 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:45141
I20260928 23:53:37.132438 25101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:45141 every 8 connection(s)
I20260928 23:53:37.133257 24973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:53:37.136255 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 24973
I20260928 23:53:37.136410 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
I20260928 23:53:37.136731 6752 external_mini_cluster.cc:1428] Setting key c3a6f6be39b15375cb32aef38ddf6347
Sep 28 23:53:37 dist-test-slave-p5bw krb5kdc[24866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639617, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.152.2@KRBTEST.COM; defaulting to no policy
I20260928 23:53:37.147115 24950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:37.134956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:58925 (local address 127.6.152.62:40121)
0928 23:53:37.135097 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:37.135102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:37.135936 (+ 834us) server_negotiation.cc:407] Connection header received
0928 23:53:37.136948 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:37.136952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:37.137008 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:37.137097 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:37.138387 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.139207 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.140362 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.140606 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.140711 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:37.143484 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:37.143487 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:37.143490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:53:37.143512 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:37.144996 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:37.145572 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:37.145575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:37.145576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:37.145648 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:37.146033 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:37.146036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:37.146037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:37.146194 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:37.146266 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:37.146889 (+ 623us) server_negotiation.cc:299] Negotiation successful
0928 23:53:37.146994 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:53:37.147927 25104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:37.135376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40121 (local address 127.6.152.1:58925)
0928 23:53:37.135786 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:37.135833 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:37.136694 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:37.137303 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:37.137313 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:37.137668 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:53:37.138246 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:37.138263 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.139351 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.139353 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:37.140235 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:37.140244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.140747 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.140751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:37.140947 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:37.141506 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:53:37.141521 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:53:37.143320 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:53:37.145110 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:53:37.145118 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:53:37.145134 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:53:37.145475 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:53:37.145737 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:53:37.145743 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:53:37.145745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:53:37.145940 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:53:37.146291 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:53:37.146297 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:53:37.146618 (+ 321us) client_negotiation.cc:769] Sending connection context
0928 23:53:37.146827 (+ 209us) client_negotiation.cc:240] Negotiation successful
0928 23:53:37.147082 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.2@KRBTEST.COM" created.
I20260928 23:53:37.149154 25102 heartbeater.cc:347] Connected to a master server at 127.6.152.62:40121
I20260928 23:53:37.149384 25102 heartbeater.cc:464] Registering TS with master...
I20260928 23:53:37.150013 25102 heartbeater.cc:511] Master 127.6.152.62:40121 requested a full tablet report, sending...
I20260928 23:53:37.151526 24911 ts_manager.cc:194] Registered new tserver with Master: a60a591f2cad4b2aab975492b6b55b47 (127.6.152.1:45141)
I20260928 23:53:37.153184 24911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.1@KRBTEST.COM'} at 127.6.152.1:58925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
I20260928 23:53:37.163585 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:40121
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:43143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:53:37.271092 25107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:37.271337 25107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:37.271399 25107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:37.275454 25107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:37.275522 25107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:37.275628 25107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:53:37.280931 25107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:40121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:37.282006 25107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:37.283028 25107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:37.292565 25112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:37.292611 25113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:37.292658 25107 server_base.cc:1034] running on GCE node
W20260928 23:53:37.292585 25115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:37.293325 25107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:37.293984 25107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:37.295166 25107 hybrid_clock.cc:648] HybridClock initialized: now 1790639617295149 us; error 27 us; skew 500 ppm
Sep 28 23:53:37 dist-test-slave-p5bw krb5kdc[24866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639617, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:37.297900 25107 init.cc:377] Logged in from keytab as kudu/127.6.152.2@KRBTEST.COM (short username kudu)
I20260928 23:53:37.298727 25107 webserver.cc:460] Webserver started at http://127.6.152.2:37253/ using document root <none> and password file <none>
I20260928 23:53:37.299222 25107 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:37.299279 25107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:37.299441 25107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:37.301529 25107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "a6d90e9139d546438a92f70edce7d6f2"
format_stamp: "Formatted at 2026-09-28 23:53:37 on dist-test-slave-p5bw"
server_key: "b2420b2a6e4fd5a5bed09a5b6dc1685b"
server_key_iv: "a32c7657b03befceed1fbdaa5bbbedaf"
server_key_version: "encryptionkey@0"
I20260928 23:53:37.302024 25107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "a6d90e9139d546438a92f70edce7d6f2"
format_stamp: "Formatted at 2026-09-28 23:53:37 on dist-test-slave-p5bw"
server_key: "b2420b2a6e4fd5a5bed09a5b6dc1685b"
server_key_iv: "a32c7657b03befceed1fbdaa5bbbedaf"
server_key_version: "encryptionkey@0"
I20260928 23:53:37.305667 25107 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:53:37.307951 25122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:37.309219 25107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:53:37.309352 25107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "a6d90e9139d546438a92f70edce7d6f2"
format_stamp: "Formatted at 2026-09-28 23:53:37 on dist-test-slave-p5bw"
server_key: "b2420b2a6e4fd5a5bed09a5b6dc1685b"
server_key_iv: "a32c7657b03befceed1fbdaa5bbbedaf"
server_key_version: "encryptionkey@0"
I20260928 23:53:37.309487 25107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:37.325330 25107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:37.328538 25107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:37.328683 25107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:37.329275 25107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:53:37.330094 25107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:53:37.330144 25107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:37.330183 25107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:53:37.330205 25107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:37.338968 25107 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:37219
I20260928 23:53:37.339012 25235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:37219 every 8 connection(s)
I20260928 23:53:37.339865 25107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:53:37.349136 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 25107
Sep 28 23:53:37 dist-test-slave-p5bw krb5kdc[24866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639617, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:53:37.349288 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:53:37.349609 6752 external_mini_cluster.cc:1428] Setting key 986821004465ff8f94fab07147eb4271
I20260928 23:53:37.353873 24950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:37.341616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:44293 (local address 127.6.152.62:40121)
0928 23:53:37.341740 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:37.341744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:37.342513 (+ 769us) server_negotiation.cc:407] Connection header received
0928 23:53:37.343833 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:37.343837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:37.343895 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:37.343995 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:37.345218 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.345752 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.346903 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.347130 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.347239 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:37.349887 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:37.349890 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:37.349892 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:53:37.349919 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:37.351628 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:37.352217 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:37.352219 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:37.352220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:37.352292 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:37.352688 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:37.352693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:37.352697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:37.352970 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:37.353080 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:37.353529 (+ 449us) server_negotiation.cc:299] Negotiation successful
0928 23:53:37.353674 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 23:53:37.354733 25238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:37.341948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40121 (local address 127.6.152.2:44293)
0928 23:53:37.342369 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:37.342412 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:37.343625 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:37.344173 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:37.344180 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:37.344526 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:53:37.345077 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:37.345091 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.345879 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.345882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:37.346773 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:37.346782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.347280 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.347284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:37.347451 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:37.348022 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:53:37.348038 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:53:37.349724 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:53:37.351736 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:53:37.351745 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:53:37.351762 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:53:37.352124 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:53:37.352375 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:53:37.352381 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:53:37.352392 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:53:37.352579 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:53:37.353071 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:53:37.353076 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:53:37.353296 (+ 220us) client_negotiation.cc:769] Sending connection context
0928 23:53:37.353458 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 23:53:37.353697 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":111,"threads_started":1}
I20260928 23:53:37.355870 25236 heartbeater.cc:347] Connected to a master server at 127.6.152.62:40121
I20260928 23:53:37.356097 25236 heartbeater.cc:464] Registering TS with master...
I20260928 23:53:37.356557 25236 heartbeater.cc:511] Master 127.6.152.62:40121 requested a full tablet report, sending...
I20260928 23:53:37.357606 24911 ts_manager.cc:194] Registered new tserver with Master: a6d90e9139d546438a92f70edce7d6f2 (127.6.152.2:37219)
I20260928 23:53:37.358146 24911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.2@KRBTEST.COM'} at 127.6.152.2:44293
WARNING: no policy specified for kudu/127.6.152.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
I20260928 23:53:37.375744 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:40121
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:43143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:53:37.490381 25241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:37.490844 25241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:37.491066 25241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:37.495339 25241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:37.495438 25241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:37.495530 25241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:53:37.500985 25241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:40121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:37.502059 25241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:37.503082 25241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:37.513106 25247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:37.513101 25249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:37.513101 25246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:37.513159 25241 server_base.cc:1034] running on GCE node
I20260928 23:53:37.513728 25241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:37.514338 25241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:37.515516 25241 hybrid_clock.cc:648] HybridClock initialized: now 1790639617515488 us; error 43 us; skew 500 ppm
Sep 28 23:53:37 dist-test-slave-p5bw krb5kdc[24866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639617, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:37.518603 25241 init.cc:377] Logged in from keytab as kudu/127.6.152.3@KRBTEST.COM (short username kudu)
I20260928 23:53:37.519450 25241 webserver.cc:460] Webserver started at http://127.6.152.3:39547/ using document root <none> and password file <none>
I20260928 23:53:37.519945 25241 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:37.520002 25241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:37.520154 25241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:37.522106 25241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "7870ce22ea5846bbb49fe4f9cd40bc9b"
format_stamp: "Formatted at 2026-09-28 23:53:37 on dist-test-slave-p5bw"
server_key: "f7e1c8720d159f95a360135af465ee9b"
server_key_iv: "ef34e36e3ed3c7fed99435331333c2f6"
server_key_version: "encryptionkey@0"
I20260928 23:53:37.522650 25241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "7870ce22ea5846bbb49fe4f9cd40bc9b"
format_stamp: "Formatted at 2026-09-28 23:53:37 on dist-test-slave-p5bw"
server_key: "f7e1c8720d159f95a360135af465ee9b"
server_key_iv: "ef34e36e3ed3c7fed99435331333c2f6"
server_key_version: "encryptionkey@0"
I20260928 23:53:37.526217 25241 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:53:37.528889 25256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:37.530128 25241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:53:37.530220 25241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "7870ce22ea5846bbb49fe4f9cd40bc9b"
format_stamp: "Formatted at 2026-09-28 23:53:37 on dist-test-slave-p5bw"
server_key: "f7e1c8720d159f95a360135af465ee9b"
server_key_iv: "ef34e36e3ed3c7fed99435331333c2f6"
server_key_version: "encryptionkey@0"
I20260928 23:53:37.530349 25241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:37.557773 25241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:37.560703 25241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:37.560926 25241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:37.561527 25241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:53:37.562326 25241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:53:37.562371 25241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:37.562459 25241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:53:37.562497 25241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:37.571094 25241 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:46637
I20260928 23:53:37.571125 25369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:46637 every 8 connection(s)
I20260928 23:53:37.571960 25241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:53:37.572180 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 25241
I20260928 23:53:37.572325 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:53:37.572676 6752 external_mini_cluster.cc:1428] Setting key ddcbe258273fb5bf894a3970de4fc4b1
Sep 28 23:53:37 dist-test-slave-p5bw krb5kdc[24866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639617, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:53:37.584936 24950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:37.573810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:52069 (local address 127.6.152.62:40121)
0928 23:53:37.573952 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:37.573959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:37.575061 (+ 1102us) server_negotiation.cc:407] Connection header received
0928 23:53:37.576490 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:37.576492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:37.576536 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:37.576604 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:37.577747 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.578273 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.579317 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.579537 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.579611 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:37.581954 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:37.581956 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:37.581958 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:53:37.581980 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:37.583150 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:37.583586 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:37.583588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:37.583589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:37.583656 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:37.583933 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:37.583935 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:37.583936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:37.584086 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:37.584157 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:37.584594 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 23:53:37.584715 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 23:53:37.585837 25372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:37.574323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:40121 (local address 127.6.152.3:52069)
0928 23:53:37.574919 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:37.574977 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:37.576307 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:37.576749 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:37.576755 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:37.577157 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:53:37.577617 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:37.577631 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.578372 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.578374 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:37.579217 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:37.579226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.579654 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.579657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:37.579787 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:37.580394 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:53:37.580411 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:53:37.581810 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:53:37.583257 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:53:37.583262 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:53:37.583272 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:53:37.583506 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:53:37.583728 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:53:37.583730 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:53:37.583731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:53:37.583852 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:53:37.584165 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:53:37.584170 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:53:37.584374 (+ 204us) client_negotiation.cc:769] Sending connection context
0928 23:53:37.584523 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 23:53:37.584766 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":154,"threads_started":1}
I20260928 23:53:37.586869 25370 heartbeater.cc:347] Connected to a master server at 127.6.152.62:40121
I20260928 23:53:37.587064 25370 heartbeater.cc:464] Registering TS with master...
I20260928 23:53:37.587463 25370 heartbeater.cc:511] Master 127.6.152.62:40121 requested a full tablet report, sending...
I20260928 23:53:37.588325 24910 ts_manager.cc:194] Registered new tserver with Master: 7870ce22ea5846bbb49fe4f9cd40bc9b (127.6.152.3:46637)
I20260928 23:53:37.588959 24910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.3@KRBTEST.COM'} at 127.6.152.3:52069
I20260928 23:53:37.598228 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:53:37.607221 24950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:37.600052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48204 (local address 127.6.152.62:40121)
0928 23:53:37.600194 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:37.600201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:37.600378 (+ 177us) server_negotiation.cc:407] Connection header received
0928 23:53:37.600481 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:37.600483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:37.600522 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:37.600583 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:37.601277 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.601764 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.602562 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.602756 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.602825 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:37.604311 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:37.604314 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:37.604316 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:53:37.604333 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:37.605664 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:37.606108 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:37.606111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:37.606112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:37.606180 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:37.606428 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:37.606430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:37.606431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:37.606613 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:37.606720 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:37.606939 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 23:53:37.607039 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:53:37.610287 24910 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:53:37.613548 24910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:53:37.629429 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:37.623625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:45141 (local address 127.0.0.1:40886)
0928 23:53:37.624306 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:37.624317 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:37.624543 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:37.625341 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:37.625343 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:37.625355 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:37.625431 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:37.625435 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.626587 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.626593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:37.627789 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:37.627798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.628863 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.628867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:37.628997 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:37.629039 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 23:53:37.629156 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 23:53:37.629272 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":64,"spinlock_wait_cycles":275584,"thread_start_us":79,"threads_started":1}
I20260928 23:53:37.629793 25384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:37.623459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48346 (local address 127.6.152.2:37219)
0928 23:53:37.623755 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:37.623774 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:37.624105 (+ 331us) server_negotiation.cc:407] Connection header received
0928 23:53:37.624644 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:37.624657 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:37.624909 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:37.625065 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:37.625352 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.626565 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.628214 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.629080 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.629205 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:37.629355 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:37.629411 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 23:53:37.629498 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":116,"threads_started":1}
I20260928 23:53:37.629599 25383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:37.623284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:37219 (local address 127.0.0.1:48346)
0928 23:53:37.624009 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:37.624021 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:37.624496 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:37.625067 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:37.625073 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:37.625096 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:37.625229 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:37.625237 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.626715 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.626718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:37.628082 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:37.628093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.629203 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.629207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:37.629294 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:37.629316 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:53:37.629380 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 23:53:37.629462 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":9,"thread_start_us":101,"threads_started":1}
I20260928 23:53:37.630096 25386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:37.623772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40886 (local address 127.6.152.1:45141)
0928 23:53:37.624096 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:37.624113 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:37.624391 (+ 278us) server_negotiation.cc:407] Connection header received
0928 23:53:37.624699 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:37.624703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:37.624925 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:37.625087 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:37.625533 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.626264 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.627913 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.628698 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.628857 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:37.629009 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:37.629699 (+ 690us) server_negotiation.cc:299] Negotiation successful
0928 23:53:37.629945 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":126,"threads_started":1}
I20260928 23:53:37.630440 25382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:37.622701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46736 (local address 127.6.152.3:46637)
0928 23:53:37.624764 (+ 2063us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:37.624774 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:37.624794 (+ 20us) server_negotiation.cc:407] Connection header received
0928 23:53:37.624881 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:37.624886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:37.625045 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:37.625170 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:37.625604 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.626685 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.628893 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.629702 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.629803 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:37.629934 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:37.630236 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 23:53:37.630328 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1965,"thread_start_us":140,"threads_started":1}
I20260928 23:53:37.630404 25381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:37.622572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:46637 (local address 127.0.0.1:46736)
0928 23:53:37.623117 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:37.623172 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:37.623305 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:37.625349 (+ 2044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:37.625351 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:37.625380 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:37.625479 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:37.625485 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.626809 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.626812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:37.628088 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:37.628094 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.629994 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.629997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:37.630088 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:37.630120 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 23:53:37.630197 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 23:53:37.630268 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":97,"spinlock_wait_cycles":384,"thread_start_us":100,"threads_started":1}
I20260928 23:53:37.633268 25036 tablet_service.cc:1467] Processing CreateTablet for tablet 62d0070c46f0420f93d835bba08837bf (DEFAULT_TABLE table=test-table [id=1822a6e8d9a3481c8d5c366aaa3ddfde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:53:37.633713 25170 tablet_service.cc:1467] Processing CreateTablet for tablet 62d0070c46f0420f93d835bba08837bf (DEFAULT_TABLE table=test-table [id=1822a6e8d9a3481c8d5c366aaa3ddfde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:53:37.633948 25304 tablet_service.cc:1467] Processing CreateTablet for tablet 62d0070c46f0420f93d835bba08837bf (DEFAULT_TABLE table=test-table [id=1822a6e8d9a3481c8d5c366aaa3ddfde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:53:37.634435 25036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d0070c46f0420f93d835bba08837bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:37.634696 25170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d0070c46f0420f93d835bba08837bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:37.634877 25304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d0070c46f0420f93d835bba08837bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:37.640293 25387 tablet_bootstrap.cc:492] T 62d0070c46f0420f93d835bba08837bf P 7870ce22ea5846bbb49fe4f9cd40bc9b: Bootstrap starting.
I20260928 23:53:37.642380 25389 tablet_bootstrap.cc:492] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2: Bootstrap starting.
I20260928 23:53:37.642771 25387 tablet_bootstrap.cc:654] T 62d0070c46f0420f93d835bba08837bf P 7870ce22ea5846bbb49fe4f9cd40bc9b: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:37.643783 25387 log.cc:824] T 62d0070c46f0420f93d835bba08837bf P 7870ce22ea5846bbb49fe4f9cd40bc9b: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:37.644387 25389 tablet_bootstrap.cc:654] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:37.645170 25388 tablet_bootstrap.cc:492] T 62d0070c46f0420f93d835bba08837bf P a60a591f2cad4b2aab975492b6b55b47: Bootstrap starting.
I20260928 23:53:37.645157 25389 log.cc:824] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:37.645620 25387 tablet_bootstrap.cc:492] T 62d0070c46f0420f93d835bba08837bf P 7870ce22ea5846bbb49fe4f9cd40bc9b: No bootstrap required, opened a new log
I20260928 23:53:37.645857 25387 ts_tablet_manager.cc:1397] T 62d0070c46f0420f93d835bba08837bf P 7870ce22ea5846bbb49fe4f9cd40bc9b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260928 23:53:37.647029 25389 tablet_bootstrap.cc:492] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2: No bootstrap required, opened a new log
I20260928 23:53:37.647043 25388 tablet_bootstrap.cc:654] T 62d0070c46f0420f93d835bba08837bf P a60a591f2cad4b2aab975492b6b55b47: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:37.647199 25389 ts_tablet_manager.cc:1397] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 23:53:37.648000 25388 log.cc:824] T 62d0070c46f0420f93d835bba08837bf P a60a591f2cad4b2aab975492b6b55b47: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:37.649732 25387 raft_consensus.cc:348] T 62d0070c46f0420f93d835bba08837bf P 7870ce22ea5846bbb49fe4f9cd40bc9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ce22ea5846bbb49fe4f9cd40bc9b" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46637 } } peers { permanent_uuid: "a6d90e9139d546438a92f70edce7d6f2" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 37219 } } peers { permanent_uuid: "a60a591f2cad4b2aab975492b6b55b47" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 45141 } }
I20260928 23:53:37.650107 25387 raft_consensus.cc:374] T 62d0070c46f0420f93d835bba08837bf P 7870ce22ea5846bbb49fe4f9cd40bc9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:37.650175 25387 raft_consensus.cc:729] T 62d0070c46f0420f93d835bba08837bf P 7870ce22ea5846bbb49fe4f9cd40bc9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7870ce22ea5846bbb49fe4f9cd40bc9b, State: Initialized, Role: FOLLOWER
I20260928 23:53:37.650352 25388 tablet_bootstrap.cc:492] T 62d0070c46f0420f93d835bba08837bf P a60a591f2cad4b2aab975492b6b55b47: No bootstrap required, opened a new log
I20260928 23:53:37.650542 25388 ts_tablet_manager.cc:1397] T 62d0070c46f0420f93d835bba08837bf P a60a591f2cad4b2aab975492b6b55b47: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:53:37.650703 25387 consensus_queue.cc:260] T 62d0070c46f0420f93d835bba08837bf P 7870ce22ea5846bbb49fe4f9cd40bc9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ce22ea5846bbb49fe4f9cd40bc9b" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46637 } } peers { permanent_uuid: "a6d90e9139d546438a92f70edce7d6f2" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 37219 } } peers { permanent_uuid: "a60a591f2cad4b2aab975492b6b55b47" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 45141 } }
I20260928 23:53:37.650981 25389 raft_consensus.cc:348] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ce22ea5846bbb49fe4f9cd40bc9b" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46637 } } peers { permanent_uuid: "a6d90e9139d546438a92f70edce7d6f2" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 37219 } } peers { permanent_uuid: "a60a591f2cad4b2aab975492b6b55b47" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 45141 } }
I20260928 23:53:37.651322 25389 raft_consensus.cc:374] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:37.651407 25389 raft_consensus.cc:729] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6d90e9139d546438a92f70edce7d6f2, State: Initialized, Role: FOLLOWER
I20260928 23:53:37.651721 25387 ts_tablet_manager.cc:1428] T 62d0070c46f0420f93d835bba08837bf P 7870ce22ea5846bbb49fe4f9cd40bc9b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:53:37.651871 25389 consensus_queue.cc:260] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ce22ea5846bbb49fe4f9cd40bc9b" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46637 } } peers { permanent_uuid: "a6d90e9139d546438a92f70edce7d6f2" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 37219 } } peers { permanent_uuid: "a60a591f2cad4b2aab975492b6b55b47" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 45141 } }
I20260928 23:53:37.652072 25370 heartbeater.cc:503] Master 127.6.152.62:40121 was elected leader, sending a full tablet report...
I20260928 23:53:37.652552 25236 heartbeater.cc:503] Master 127.6.152.62:40121 was elected leader, sending a full tablet report...
I20260928 23:53:37.652858 25389 ts_tablet_manager.cc:1428] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 23:53:37.654289 25388 raft_consensus.cc:348] T 62d0070c46f0420f93d835bba08837bf P a60a591f2cad4b2aab975492b6b55b47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ce22ea5846bbb49fe4f9cd40bc9b" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46637 } } peers { permanent_uuid: "a6d90e9139d546438a92f70edce7d6f2" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 37219 } } peers { permanent_uuid: "a60a591f2cad4b2aab975492b6b55b47" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 45141 } }
I20260928 23:53:37.654552 25388 raft_consensus.cc:374] T 62d0070c46f0420f93d835bba08837bf P a60a591f2cad4b2aab975492b6b55b47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:37.654646 25388 raft_consensus.cc:729] T 62d0070c46f0420f93d835bba08837bf P a60a591f2cad4b2aab975492b6b55b47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60a591f2cad4b2aab975492b6b55b47, State: Initialized, Role: FOLLOWER
I20260928 23:53:37.655093 25388 consensus_queue.cc:260] T 62d0070c46f0420f93d835bba08837bf P a60a591f2cad4b2aab975492b6b55b47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ce22ea5846bbb49fe4f9cd40bc9b" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46637 } } peers { permanent_uuid: "a6d90e9139d546438a92f70edce7d6f2" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 37219 } } peers { permanent_uuid: "a60a591f2cad4b2aab975492b6b55b47" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 45141 } }
I20260928 23:53:37.655634 25102 heartbeater.cc:503] Master 127.6.152.62:40121 was elected leader, sending a full tablet report...
I20260928 23:53:37.655988 25388 ts_tablet_manager.cc:1428] T 62d0070c46f0420f93d835bba08837bf P a60a591f2cad4b2aab975492b6b55b47: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 23:53:37.678099 25394 raft_consensus.cc:482] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:53:37.678308 25394 raft_consensus.cc:504] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ce22ea5846bbb49fe4f9cd40bc9b" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46637 } } peers { permanent_uuid: "a6d90e9139d546438a92f70edce7d6f2" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 37219 } } peers { permanent_uuid: "a60a591f2cad4b2aab975492b6b55b47" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 45141 } }
I20260928 23:53:37.679391 25394 leader_election.cc:288] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7870ce22ea5846bbb49fe4f9cd40bc9b (127.6.152.3:46637), a60a591f2cad4b2aab975492b6b55b47 (127.6.152.1:45141)
I20260928 23:53:37.684137 25382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:37.679688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:43357 (local address 127.6.152.3:46637)
0928 23:53:37.679805 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:37.679812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:37.680097 (+ 285us) server_negotiation.cc:407] Connection header received
0928 23:53:37.680188 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:37.680191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:37.680243 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:37.680301 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:37.680587 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.681320 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.682863 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.683467 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.683576 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:37.683635 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:37.683944 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 23:53:37.684013 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:53:37.684137 25386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:37.679724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:46705 (local address 127.6.152.1:45141)
0928 23:53:37.679845 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:37.679849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:37.679865 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:53:37.679993 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:37.679995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:37.680033 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:37.680132 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:37.680431 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.681239 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.682863 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.683467 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.683576 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:37.683635 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:37.683944 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 23:53:37.684013 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:53:37.684186 25238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:37.679597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:45141 (local address 127.6.152.2:46705)
0928 23:53:37.679770 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:37.679786 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:37.679870 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:37.680163 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:37.680165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:37.680182 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:37.680311 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:37.680319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.681348 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.681350 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:37.682748 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:37.682754 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.683586 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.683590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:37.683796 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:37.683835 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 23:53:37.683902 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 23:53:37.684047 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":219}
I20260928 23:53:37.684195 25396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:37.679599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:46637 (local address 127.6.152.2:43357)
0928 23:53:37.680000 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:37.680018 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:37.680087 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:37.680336 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:37.680340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:37.680360 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:37.680480 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:37.680484 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.681425 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.681428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:37.682738 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:37.682750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.683586 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.683590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:37.683756 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:37.683823 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 23:53:37.683902 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 23:53:37.684039 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":163,"thread_start_us":125,"threads_started":1}
I20260928 23:53:37.684796 25056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d0070c46f0420f93d835bba08837bf" candidate_uuid: "a6d90e9139d546438a92f70edce7d6f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60a591f2cad4b2aab975492b6b55b47" is_pre_election: true
I20260928 23:53:37.685144 25056 raft_consensus.cc:2454] T 62d0070c46f0420f93d835bba08837bf P a60a591f2cad4b2aab975492b6b55b47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6d90e9139d546438a92f70edce7d6f2 in term 0.
I20260928 23:53:37.685278 25324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d0070c46f0420f93d835bba08837bf" candidate_uuid: "a6d90e9139d546438a92f70edce7d6f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7870ce22ea5846bbb49fe4f9cd40bc9b" is_pre_election: true
I20260928 23:53:37.685544 25324 raft_consensus.cc:2454] T 62d0070c46f0420f93d835bba08837bf P 7870ce22ea5846bbb49fe4f9cd40bc9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6d90e9139d546438a92f70edce7d6f2 in term 0.
I20260928 23:53:37.685634 25126 leader_election.cc:302] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a60a591f2cad4b2aab975492b6b55b47, a6d90e9139d546438a92f70edce7d6f2; no voters:
I20260928 23:53:37.685936 25394 raft_consensus.cc:2781] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:53:37.686019 25394 raft_consensus.cc:482] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:53:37.686060 25394 raft_consensus.cc:3037] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:37.687044 25394 raft_consensus.cc:504] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ce22ea5846bbb49fe4f9cd40bc9b" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46637 } } peers { permanent_uuid: "a6d90e9139d546438a92f70edce7d6f2" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 37219 } } peers { permanent_uuid: "a60a591f2cad4b2aab975492b6b55b47" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 45141 } }
I20260928 23:53:37.687355 25394 leader_election.cc:288] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2 [CANDIDATE]: Term 1 election: Requested vote from peers 7870ce22ea5846bbb49fe4f9cd40bc9b (127.6.152.3:46637), a60a591f2cad4b2aab975492b6b55b47 (127.6.152.1:45141)
I20260928 23:53:37.687625 25324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d0070c46f0420f93d835bba08837bf" candidate_uuid: "a6d90e9139d546438a92f70edce7d6f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7870ce22ea5846bbb49fe4f9cd40bc9b"
I20260928 23:53:37.687660 25056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d0070c46f0420f93d835bba08837bf" candidate_uuid: "a6d90e9139d546438a92f70edce7d6f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60a591f2cad4b2aab975492b6b55b47"
I20260928 23:53:37.687773 25324 raft_consensus.cc:3037] T 62d0070c46f0420f93d835bba08837bf P 7870ce22ea5846bbb49fe4f9cd40bc9b [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:37.687819 25056 raft_consensus.cc:3037] T 62d0070c46f0420f93d835bba08837bf P a60a591f2cad4b2aab975492b6b55b47 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:37.688784 25324 raft_consensus.cc:2454] T 62d0070c46f0420f93d835bba08837bf P 7870ce22ea5846bbb49fe4f9cd40bc9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6d90e9139d546438a92f70edce7d6f2 in term 1.
I20260928 23:53:37.688943 25056 raft_consensus.cc:2454] T 62d0070c46f0420f93d835bba08837bf P a60a591f2cad4b2aab975492b6b55b47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6d90e9139d546438a92f70edce7d6f2 in term 1.
I20260928 23:53:37.689093 25125 leader_election.cc:302] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7870ce22ea5846bbb49fe4f9cd40bc9b, a6d90e9139d546438a92f70edce7d6f2; no voters:
I20260928 23:53:37.689256 25394 raft_consensus.cc:2781] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:53:37.689481 25394 raft_consensus.cc:686] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2 [term 1 LEADER]: Becoming Leader. State: Replica: a6d90e9139d546438a92f70edce7d6f2, State: Running, Role: LEADER
I20260928 23:53:37.689836 25394 consensus_queue.cc:237] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ce22ea5846bbb49fe4f9cd40bc9b" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46637 } } peers { permanent_uuid: "a6d90e9139d546438a92f70edce7d6f2" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 37219 } } peers { permanent_uuid: "a60a591f2cad4b2aab975492b6b55b47" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 45141 } }
I20260928 23:53:37.692752 24911 catalog_manager.cc:5495] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6d90e9139d546438a92f70edce7d6f2 (127.6.152.2). New cstate: current_term: 1 leader_uuid: "a6d90e9139d546438a92f70edce7d6f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ce22ea5846bbb49fe4f9cd40bc9b" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6d90e9139d546438a92f70edce7d6f2" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 37219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a60a591f2cad4b2aab975492b6b55b47" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 45141 } health_report { overall_health: UNKNOWN } } }
I20260928 23:53:37.713414 25384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:37.709900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48360 (local address 127.6.152.2:37219)
0928 23:53:37.710055 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:37.710058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:37.710072 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:53:37.710242 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:37.710245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:37.710293 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:37.710377 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:53:37.710661 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.711192 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.711991 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.712189 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.712262 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:37.712572 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:53:37.712981 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:53:37.713091 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:37.713182 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 23:53:37.713212 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 23:53:37.719197 25324 raft_consensus.cc:1261] T 62d0070c46f0420f93d835bba08837bf P 7870ce22ea5846bbb49fe4f9cd40bc9b [term 1 FOLLOWER]: Refusing update from remote peer a6d90e9139d546438a92f70edce7d6f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:53:37.719197 25056 raft_consensus.cc:1261] T 62d0070c46f0420f93d835bba08837bf P a60a591f2cad4b2aab975492b6b55b47 [term 1 FOLLOWER]: Refusing update from remote peer a6d90e9139d546438a92f70edce7d6f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:53:37.719899 25397 consensus_queue.cc:1035] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7870ce22ea5846bbb49fe4f9cd40bc9b" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 46637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:53:37.720105 25394 consensus_queue.cc:1035] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a60a591f2cad4b2aab975492b6b55b47" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:53:37.726790 25400 mvcc.cc:205] Tried to move back new op lower bound from 7334459874170556416 to 7334459874060177408. Current Snapshot: MvccSnapshot[applied={T|T < 7334459874170556416}]
I20260928 23:53:37.727655 25401 mvcc.cc:205] Tried to move back new op lower bound from 7334459874170556416 to 7334459874060177408. Current Snapshot: MvccSnapshot[applied={T|T < 7334459874170556416}]
I20260928 23:53:37.734948 24910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:53:37.735122 24910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:53:37.737780 24910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c819fa23773746daa90449cc5c82b09e: Sending DeleteTablet for 3 replicas of tablet 62d0070c46f0420f93d835bba08837bf
I20260928 23:53:37.738514 25304 tablet_service.cc:1514] Processing DeleteTablet for tablet 62d0070c46f0420f93d835bba08837bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:53:37 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:46736
I20260928 23:53:37.738548 25170 tablet_service.cc:1514] Processing DeleteTablet for tablet 62d0070c46f0420f93d835bba08837bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:53:37 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:48346
I20260928 23:53:37.738706 25036 tablet_service.cc:1514] Processing DeleteTablet for tablet 62d0070c46f0420f93d835bba08837bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:53:37 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:40886
I20260928 23:53:37.739099 25410 tablet_replica.cc:330] T 62d0070c46f0420f93d835bba08837bf P 7870ce22ea5846bbb49fe4f9cd40bc9b: stopping tablet replica
I20260928 23:53:37.739272 25411 tablet_replica.cc:330] T 62d0070c46f0420f93d835bba08837bf P a60a591f2cad4b2aab975492b6b55b47: stopping tablet replica
I20260928 23:53:37.739377 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 24973
I20260928 23:53:37.745502 25409 tablet_replica.cc:330] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2: stopping tablet replica
I20260928 23:53:37.745536 25410 raft_consensus.cc:2229] T 62d0070c46f0420f93d835bba08837bf P 7870ce22ea5846bbb49fe4f9cd40bc9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:53:37.745817 25409 raft_consensus.cc:2229] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:53:37.745894 25410 raft_consensus.cc:2258] T 62d0070c46f0420f93d835bba08837bf P 7870ce22ea5846bbb49fe4f9cd40bc9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:53:37.746367 25409 raft_consensus.cc:2258] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 23:53:37.746626 24898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.152.1:45141 (error 108)
I20260928 23:53:37.747362 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 25107
I20260928 23:53:37.747448 25410 ts_tablet_manager.cc:1905] T 62d0070c46f0420f93d835bba08837bf P 7870ce22ea5846bbb49fe4f9cd40bc9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:53:37.747803 25409 ts_tablet_manager.cc:1905] T 62d0070c46f0420f93d835bba08837bf P a6d90e9139d546438a92f70edce7d6f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:53:37.748613 25381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:37.748283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:45141 (local address 127.0.0.1:40900)
0928 23:53:37.748431 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:37.748514 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.1:45141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 23:53:37.749413 24898 catalog_manager.cc:4584] TS a60a591f2cad4b2aab975492b6b55b47 (127.6.152.1:45141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62d0070c46f0420f93d835bba08837bf: Network error: Client connection negotiation failed: client connection to 127.6.152.1:45141: connect: Connection refused (error 111)
I20260928 23:53:37.751147 25410 ts_tablet_manager.cc:1918] T 62d0070c46f0420f93d835bba08837bf P 7870ce22ea5846bbb49fe4f9cd40bc9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:53:37.751246 25410 log.cc:1196] T 62d0070c46f0420f93d835bba08837bf P 7870ce22ea5846bbb49fe4f9cd40bc9b: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/62d0070c46f0420f93d835bba08837bf
I20260928 23:53:37.751611 25410 ts_tablet_manager.cc:1939] T 62d0070c46f0420f93d835bba08837bf P 7870ce22ea5846bbb49fe4f9cd40bc9b: Deleting consensus metadata
I20260928 23:53:37.752852 24897 catalog_manager.cc:4841] TS 7870ce22ea5846bbb49fe4f9cd40bc9b (127.6.152.3:46637): tablet 62d0070c46f0420f93d835bba08837bf (table test-table [id=1822a6e8d9a3481c8d5c366aaa3ddfde]) successfully deleted
I20260928 23:53:37.754441 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 25241
I20260928 23:53:37.754822 25381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:37.754508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:37219 (local address 127.0.0.1:48376)
0928 23:53:37.754690 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:37.754741 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.2:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 23:53:37.759768 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 24879
2026-09-28T23:53:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2258 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T23:53:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:53:37Z Disabled control of system clock
I20260928 23:53:37.789511 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:33713
--webserver_interface=127.6.152.62
--webserver_port=0
--builtin_ntp_servers=127.6.152.20:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:33713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:53:37.898368 25416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:37.898639 25416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:37.898710 25416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:37.902717 25416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:53:37.902769 25416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:37.902784 25416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:37.902801 25416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:53:37.902858 25416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:53:37.907518 25416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:33713
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:33713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:37.908550 25416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:37.909879 25416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:37.918074 25421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:37.918113 25422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:37.918100 25424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:37.918282 25416 server_base.cc:1034] running on GCE node
I20260928 23:53:37.918892 25416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:37.919960 25416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:37.921113 25416 hybrid_clock.cc:648] HybridClock initialized: now 1790639617921104 us; error 27 us; skew 500 ppm
I20260928 23:53:37.922837 25416 webserver.cc:460] Webserver started at http://127.6.152.62:35309/ using document root <none> and password file <none>
I20260928 23:53:37.923410 25416 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:37.923470 25416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:37.923686 25416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:37.925767 25416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "a046260454a0422096b2de7fbd3ecfa6"
format_stamp: "Formatted at 2026-09-28 23:53:37 on dist-test-slave-p5bw"
server_key: "027ac118893562c2d33f175bc8b3e201"
server_key_iv: "1a0854547c1f6da4c760342763efd2a3"
server_key_version: "encryptionkey@0"
I20260928 23:53:37.926326 25416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "a046260454a0422096b2de7fbd3ecfa6"
format_stamp: "Formatted at 2026-09-28 23:53:37 on dist-test-slave-p5bw"
server_key: "027ac118893562c2d33f175bc8b3e201"
server_key_iv: "1a0854547c1f6da4c760342763efd2a3"
server_key_version: "encryptionkey@0"
I20260928 23:53:37.929813 25416 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:53:37.932062 25430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:37.933320 25416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:53:37.933465 25416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "a046260454a0422096b2de7fbd3ecfa6"
format_stamp: "Formatted at 2026-09-28 23:53:37 on dist-test-slave-p5bw"
server_key: "027ac118893562c2d33f175bc8b3e201"
server_key_iv: "1a0854547c1f6da4c760342763efd2a3"
server_key_version: "encryptionkey@0"
I20260928 23:53:37.933574 25416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:37.946259 25416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:37.946868 25416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:37.947063 25416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:37.953469 25482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:33713 every 8 connection(s)
I20260928 23:53:37.953451 25416 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:33713
I20260928 23:53:37.954447 25416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:53:37.958079 25483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:37.963481 25483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a046260454a0422096b2de7fbd3ecfa6: Bootstrap starting.
I20260928 23:53:37.964705 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 25416
I20260928 23:53:37.964854 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:53:37.965114 6752 external_mini_cluster.cc:1428] Setting key 2850eb32a31f48e8f9153d71e299c82b
I20260928 23:53:37.966164 25483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a046260454a0422096b2de7fbd3ecfa6: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:37.966988 25483 log.cc:824] T 00000000000000000000000000000000 P a046260454a0422096b2de7fbd3ecfa6: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:37.968883 25483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a046260454a0422096b2de7fbd3ecfa6: No bootstrap required, opened a new log
I20260928 23:53:37.972368 25483 raft_consensus.cc:348] T 00000000000000000000000000000000 P a046260454a0422096b2de7fbd3ecfa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a046260454a0422096b2de7fbd3ecfa6" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 33713 } }
I20260928 23:53:37.972573 25483 raft_consensus.cc:374] T 00000000000000000000000000000000 P a046260454a0422096b2de7fbd3ecfa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:37.972635 25483 raft_consensus.cc:729] T 00000000000000000000000000000000 P a046260454a0422096b2de7fbd3ecfa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a046260454a0422096b2de7fbd3ecfa6, State: Initialized, Role: FOLLOWER
I20260928 23:53:37.973202 25483 consensus_queue.cc:260] T 00000000000000000000000000000000 P a046260454a0422096b2de7fbd3ecfa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a046260454a0422096b2de7fbd3ecfa6" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 33713 } }
I20260928 23:53:37.973353 25483 raft_consensus.cc:388] T 00000000000000000000000000000000 P a046260454a0422096b2de7fbd3ecfa6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:53:37.973439 25483 raft_consensus.cc:482] T 00000000000000000000000000000000 P a046260454a0422096b2de7fbd3ecfa6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:53:37.973539 25483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a046260454a0422096b2de7fbd3ecfa6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:37.974098 25486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:37.966253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39272 (local address 127.6.152.62:33713)
0928 23:53:37.966895 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:37.966909 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:37.966941 (+ 32us) server_negotiation.cc:407] Connection header received
0928 23:53:37.967600 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:37.967616 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:37.967929 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:37.968254 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:37.968697 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.969737 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.970638 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:37.971110 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:37.971281 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:37.972139 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:37.972144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:37.972156 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:37.972535 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:37.972624 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:37.972655 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:37.972837 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:37.972985 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 23:53:37.973233 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":146,"threads_started":1}
I20260928 23:53:37.974509 25483 raft_consensus.cc:504] T 00000000000000000000000000000000 P a046260454a0422096b2de7fbd3ecfa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a046260454a0422096b2de7fbd3ecfa6" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 33713 } }
I20260928 23:53:37.974841 25483 leader_election.cc:302] T 00000000000000000000000000000000 P a046260454a0422096b2de7fbd3ecfa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a046260454a0422096b2de7fbd3ecfa6; no voters:
I20260928 23:53:37.975160 25483 leader_election.cc:288] T 00000000000000000000000000000000 P a046260454a0422096b2de7fbd3ecfa6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:53:37.975220 25488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a046260454a0422096b2de7fbd3ecfa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:53:37.975430 25488 raft_consensus.cc:686] T 00000000000000000000000000000000 P a046260454a0422096b2de7fbd3ecfa6 [term 1 LEADER]: Becoming Leader. State: Replica: a046260454a0422096b2de7fbd3ecfa6, State: Running, Role: LEADER
I20260928 23:53:37.975713 25488 consensus_queue.cc:237] T 00000000000000000000000000000000 P a046260454a0422096b2de7fbd3ecfa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a046260454a0422096b2de7fbd3ecfa6" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 33713 } }
I20260928 23:53:37.976403 25483 sys_catalog.cc:564] T 00000000000000000000000000000000 P a046260454a0422096b2de7fbd3ecfa6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:53:37.976873 25488 sys_catalog.cc:455] T 00000000000000000000000000000000 P a046260454a0422096b2de7fbd3ecfa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a046260454a0422096b2de7fbd3ecfa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a046260454a0422096b2de7fbd3ecfa6" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 33713 } } }
I20260928 23:53:37.976989 25488 sys_catalog.cc:458] T 00000000000000000000000000000000 P a046260454a0422096b2de7fbd3ecfa6 [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:37.977319 25490 sys_catalog.cc:455] T 00000000000000000000000000000000 P a046260454a0422096b2de7fbd3ecfa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a046260454a0422096b2de7fbd3ecfa6. Latest consensus state: current_term: 1 leader_uuid: "a046260454a0422096b2de7fbd3ecfa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a046260454a0422096b2de7fbd3ecfa6" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 33713 } } }
I20260928 23:53:37.977433 25492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:53:37.977451 25490 sys_catalog.cc:458] T 00000000000000000000000000000000 P a046260454a0422096b2de7fbd3ecfa6 [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:37.980334 25492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:53:37.985725 25492 catalog_manager.cc:1347] Generated new cluster ID: 0f378acbc5724c8c948082c50cad87c7
I20260928 23:53:37.985795 25492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:53:38.007332 25492 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:53:38.008563 25492 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:53:38.015027 25492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a046260454a0422096b2de7fbd3ecfa6: Generated new TSK 0
I20260928 23:53:38.015801 25492 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:53:38.027316 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:33713
--builtin_ntp_servers=127.6.152.20:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:53:38.144261 25507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:38.144495 25507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:38.144532 25507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:38.148470 25507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:38.148520 25507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:38.148602 25507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:53:38.153319 25507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:33713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:38.154376 25507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:38.155735 25507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:38.164897 25515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:38.164927 25512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:38.164994 25507 server_base.cc:1034] running on GCE node
W20260928 23:53:38.164908 25513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:38.165509 25507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:38.166126 25507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:38.167301 25507 hybrid_clock.cc:648] HybridClock initialized: now 1790639618167283 us; error 28 us; skew 500 ppm
I20260928 23:53:38.169011 25507 webserver.cc:460] Webserver started at http://127.6.152.1:45575/ using document root <none> and password file <none>
I20260928 23:53:38.169510 25507 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:38.169564 25507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:38.169713 25507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:38.171551 25507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "124c0e87339e45cd87337735e6df381a"
format_stamp: "Formatted at 2026-09-28 23:53:38 on dist-test-slave-p5bw"
server_key: "3fe532c6b256fd287ea5b03e83cc9d47"
server_key_iv: "3e0464a9d6c3a7438a6f338adadabc03"
server_key_version: "encryptionkey@0"
I20260928 23:53:38.172039 25507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "124c0e87339e45cd87337735e6df381a"
format_stamp: "Formatted at 2026-09-28 23:53:38 on dist-test-slave-p5bw"
server_key: "3fe532c6b256fd287ea5b03e83cc9d47"
server_key_iv: "3e0464a9d6c3a7438a6f338adadabc03"
server_key_version: "encryptionkey@0"
I20260928 23:53:38.175573 25507 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 23:53:38.177902 25521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:38.179076 25507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:53:38.179186 25507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "124c0e87339e45cd87337735e6df381a"
format_stamp: "Formatted at 2026-09-28 23:53:38 on dist-test-slave-p5bw"
server_key: "3fe532c6b256fd287ea5b03e83cc9d47"
server_key_iv: "3e0464a9d6c3a7438a6f338adadabc03"
server_key_version: "encryptionkey@0"
I20260928 23:53:38.179282 25507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:38.203019 25507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:38.203670 25507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:38.203873 25507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:38.204380 25507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:53:38.205233 25507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:53:38.205277 25507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:38.205315 25507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:53:38.205330 25507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:38.213871 25507 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:37715
I20260928 23:53:38.213918 25634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:37715 every 8 connection(s)
I20260928 23:53:38.214777 25507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:53:38.223639 25486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:38.216534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:48969 (local address 127.6.152.62:33713)
0928 23:53:38.216665 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:38.216669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:38.217678 (+ 1009us) server_negotiation.cc:407] Connection header received
0928 23:53:38.219047 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:38.219050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:38.219104 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:38.219182 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:38.220016 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.220563 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.221739 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.221971 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.222080 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:38.223135 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:38.223142 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:38.223145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:38.223156 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:38.223173 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:38.223189 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:38.223269 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:38.223449 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 23:53:38.223504 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:53:38.224366 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 25507
I20260928 23:53:38.224419 25637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:38.216958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:33713 (local address 127.6.152.1:48969)
0928 23:53:38.217539 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:38.217577 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:38.218834 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:38.219340 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:38.219347 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:38.219426 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:53:38.219869 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:38.219882 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.220701 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.220704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:38.221607 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:38.221616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.222113 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.222117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:38.222273 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:38.222894 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:53:38.222896 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:53:38.222933 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:53:38.222988 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:53:38.223284 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:53:38.223295 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:53:38.223307 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 23:53:38.223434 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 23:53:38.223700 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":102,"threads_started":1}
I20260928 23:53:38.224501 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
I20260928 23:53:38.224819 6752 external_mini_cluster.cc:1428] Setting key 15cf18ec987cd702548f9a14a9e6b76d
I20260928 23:53:38.225706 25635 heartbeater.cc:347] Connected to a master server at 127.6.152.62:33713
I20260928 23:53:38.225924 25635 heartbeater.cc:464] Registering TS with master...
I20260928 23:53:38.226431 25635 heartbeater.cc:511] Master 127.6.152.62:33713 requested a full tablet report, sending...
I20260928 23:53:38.226769 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:33713
--builtin_ntp_servers=127.6.152.20:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 23:53:38.227820 25447 ts_manager.cc:194] Registered new tserver with Master: 124c0e87339e45cd87337735e6df381a (127.6.152.1:37715)
I20260928 23:53:38.229017 25447 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.152.1:48969
W20260928 23:53:38.333247 25638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:38.333523 25638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:38.333595 25638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:38.337678 25638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:38.337747 25638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:38.337837 25638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:53:38.342480 25638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:33713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:38.343508 25638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:38.344884 25638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:53:38.354274 25638 server_base.cc:1034] running on GCE node
W20260928 23:53:38.354207 25646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:38.354218 25643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:38.354292 25644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:38.354769 25638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:38.355368 25638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:38.356527 25638 hybrid_clock.cc:648] HybridClock initialized: now 1790639618356510 us; error 28 us; skew 500 ppm
I20260928 23:53:38.358279 25638 webserver.cc:460] Webserver started at http://127.6.152.2:34099/ using document root <none> and password file <none>
I20260928 23:53:38.358822 25638 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:38.358881 25638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:38.359109 25638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:38.361176 25638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "3f3595becacc40baab8f3708f1c87a15"
format_stamp: "Formatted at 2026-09-28 23:53:38 on dist-test-slave-p5bw"
server_key: "fdb0b94cfbbfaba6221ecbf4764be100"
server_key_iv: "5aa64ae613493130f29133ab685023f6"
server_key_version: "encryptionkey@0"
I20260928 23:53:38.361797 25638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "3f3595becacc40baab8f3708f1c87a15"
format_stamp: "Formatted at 2026-09-28 23:53:38 on dist-test-slave-p5bw"
server_key: "fdb0b94cfbbfaba6221ecbf4764be100"
server_key_iv: "5aa64ae613493130f29133ab685023f6"
server_key_version: "encryptionkey@0"
I20260928 23:53:38.365235 25638 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:53:38.367471 25652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:38.368690 25638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:53:38.368865 25638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "3f3595becacc40baab8f3708f1c87a15"
format_stamp: "Formatted at 2026-09-28 23:53:38 on dist-test-slave-p5bw"
server_key: "fdb0b94cfbbfaba6221ecbf4764be100"
server_key_iv: "5aa64ae613493130f29133ab685023f6"
server_key_version: "encryptionkey@0"
I20260928 23:53:38.368981 25638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:38.393988 25638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:38.394671 25638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:38.394868 25638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:38.395465 25638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:53:38.396373 25638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:53:38.396446 25638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:38.396518 25638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:53:38.396560 25638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:38.404933 25638 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:44683
I20260928 23:53:38.405006 25765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:44683 every 8 connection(s)
I20260928 23:53:38.405855 25638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:53:38.413672 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 25638
I20260928 23:53:38.413787 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:53:38.414077 6752 external_mini_cluster.cc:1428] Setting key d79a9366d195818c0834e1de5c61cb2a
I20260928 23:53:38.414777 25486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:38.407641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:53157 (local address 127.6.152.62:33713)
0928 23:53:38.407773 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:38.407777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:38.408771 (+ 994us) server_negotiation.cc:407] Connection header received
0928 23:53:38.410140 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:38.410143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:38.410192 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:38.410281 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:38.411256 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.411773 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.412884 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.413090 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.413167 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:38.414239 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:38.414242 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:38.414244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:38.414256 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:38.414273 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:38.414280 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:38.414356 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:38.414565 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 23:53:38.414641 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 23:53:38.415702 25768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:38.407999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:33713 (local address 127.6.152.2:53157)
0928 23:53:38.408616 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:38.408656 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:38.409953 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:38.410507 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:38.410515 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:38.410631 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:53:38.411127 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:38.411140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.411874 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.411877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:38.412729 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:38.412737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.413216 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.413219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:38.413356 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:38.413979 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:53:38.413981 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:53:38.414019 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:53:38.414078 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:53:38.414371 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:53:38.414381 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:53:38.414395 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 23:53:38.414536 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 23:53:38.414800 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":106,"threads_started":1}
I20260928 23:53:38.416169 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:33713
--builtin_ntp_servers=127.6.152.20:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 23:53:38.416785 25766 heartbeater.cc:347] Connected to a master server at 127.6.152.62:33713
I20260928 23:53:38.417057 25766 heartbeater.cc:464] Registering TS with master...
I20260928 23:53:38.417534 25766 heartbeater.cc:511] Master 127.6.152.62:33713 requested a full tablet report, sending...
I20260928 23:53:38.418516 25447 ts_manager.cc:194] Registered new tserver with Master: 3f3595becacc40baab8f3708f1c87a15 (127.6.152.2:44683)
I20260928 23:53:38.419061 25447 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.152.2:53157
W20260928 23:53:38.525051 25769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:38.525275 25769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:38.525336 25769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:38.529281 25769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:38.529326 25769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:38.529414 25769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:53:38.534104 25769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:33713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:38.535037 25769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:38.536329 25769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:38.545560 25775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:38.545552 25777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:38.545553 25774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:38.545897 25769 server_base.cc:1034] running on GCE node
I20260928 23:53:38.546281 25769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:38.546868 25769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:38.548029 25769 hybrid_clock.cc:648] HybridClock initialized: now 1790639618548014 us; error 26 us; skew 500 ppm
I20260928 23:53:38.553079 25769 webserver.cc:460] Webserver started at http://127.6.152.3:32995/ using document root <none> and password file <none>
I20260928 23:53:38.553620 25769 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:38.553679 25769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:38.553933 25769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:38.555871 25769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "a5b43815cd6b4d6d8e141be260a92a16"
format_stamp: "Formatted at 2026-09-28 23:53:38 on dist-test-slave-p5bw"
server_key: "7c0ff627f3b02acdc641c366428411f0"
server_key_iv: "023818186b2bf8fee7af91cc4bd816d0"
server_key_version: "encryptionkey@0"
I20260928 23:53:38.556416 25769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "a5b43815cd6b4d6d8e141be260a92a16"
format_stamp: "Formatted at 2026-09-28 23:53:38 on dist-test-slave-p5bw"
server_key: "7c0ff627f3b02acdc641c366428411f0"
server_key_iv: "023818186b2bf8fee7af91cc4bd816d0"
server_key_version: "encryptionkey@0"
I20260928 23:53:38.559798 25769 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 23:53:38.562155 25783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:38.563311 25769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:53:38.563421 25769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "a5b43815cd6b4d6d8e141be260a92a16"
format_stamp: "Formatted at 2026-09-28 23:53:38 on dist-test-slave-p5bw"
server_key: "7c0ff627f3b02acdc641c366428411f0"
server_key_iv: "023818186b2bf8fee7af91cc4bd816d0"
server_key_version: "encryptionkey@0"
I20260928 23:53:38.563519 25769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:38.579530 25769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:38.580171 25769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:38.580353 25769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:38.580987 25769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:53:38.581856 25769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:53:38.581907 25769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:38.581995 25769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:53:38.582029 25769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:38.590503 25769 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:42553
I20260928 23:53:38.590526 25896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:42553 every 8 connection(s)
I20260928 23:53:38.591351 25769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:53:38.592221 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 25769
I20260928 23:53:38.592300 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:53:38.592531 6752 external_mini_cluster.cc:1428] Setting key 5625dc0dd99a00e7ec6be94c68ae3bda
I20260928 23:53:38.600581 25486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:38.593870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:57863 (local address 127.6.152.62:33713)
0928 23:53:38.594015 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:38.594020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:38.595112 (+ 1092us) server_negotiation.cc:407] Connection header received
0928 23:53:38.596417 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:38.596419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:38.596460 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:38.596523 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:38.597378 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.597886 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.598930 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.599132 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.599202 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:38.600221 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:38.600224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:38.600226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:38.600236 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:38.600250 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:38.600256 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:38.600306 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:38.600439 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 23:53:38.600479 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 23:53:38.601400 25899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:38.594297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:33713 (local address 127.6.152.3:57863)
0928 23:53:38.594966 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:38.595014 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:38.596246 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:38.596646 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:38.596653 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:38.596731 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:53:38.597266 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:38.597278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.597983 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.597986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:38.598831 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:38.598839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.599247 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.599250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:38.599379 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:38.600005 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:53:38.600007 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:53:38.600044 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:53:38.600102 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:53:38.600323 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:53:38.600329 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:53:38.600333 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:53:38.600429 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 23:53:38.600631 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":132,"threads_started":1}
I20260928 23:53:38.602381 25897 heartbeater.cc:347] Connected to a master server at 127.6.152.62:33713
I20260928 23:53:38.602607 25897 heartbeater.cc:464] Registering TS with master...
I20260928 23:53:38.603082 25897 heartbeater.cc:511] Master 127.6.152.62:33713 requested a full tablet report, sending...
I20260928 23:53:38.604004 25447 ts_manager.cc:194] Registered new tserver with Master: a5b43815cd6b4d6d8e141be260a92a16 (127.6.152.3:42553)
I20260928 23:53:38.604552 25447 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.152.3:57863
I20260928 23:53:38.606434 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:53:38.612038 25486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:38.608232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39286 (local address 127.6.152.62:33713)
0928 23:53:38.608346 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:38.608349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:38.608563 (+ 214us) server_negotiation.cc:407] Connection header received
0928 23:53:38.608700 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:38.608703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:38.608748 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:38.608844 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:38.609117 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.609609 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.610430 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.610658 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.610760 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:38.611547 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:38.611551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:38.611555 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:38.611571 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:38.611587 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:38.611595 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:38.611719 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:38.611871 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 23:53:38.611906 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 23:53:38.614579 25447 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:53:38.617487 25447 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:53:38.630774 25908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:38.624971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:37715 (local address 127.0.0.1:54478)
0928 23:53:38.625747 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:38.625794 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:38.625927 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:38.626694 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:38.626699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:38.626721 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:38.626807 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:38.626814 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.627851 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.627854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:38.629059 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:38.629069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.630240 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.630243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:38.630350 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:38.630387 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 23:53:38.630483 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 23:53:38.630612 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":50,"thread_start_us":95,"threads_started":1}
I20260928 23:53:38.630774 25909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:38.625102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:44683 (local address 127.0.0.1:51594)
0928 23:53:38.625941 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:38.625954 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:38.626044 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:38.626546 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:38.626552 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:38.626580 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:38.626742 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:38.626750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.628142 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.628145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:38.629352 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:38.629360 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.630343 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.630347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:38.630442 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:38.630465 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:53:38.630518 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 23:53:38.630621 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":61,"thread_start_us":101,"threads_started":1}
I20260928 23:53:38.630805 25911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:38.625378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54478 (local address 127.6.152.1:37715)
0928 23:53:38.625807 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:38.625819 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:38.626078 (+ 259us) server_negotiation.cc:407] Connection header received
0928 23:53:38.626159 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:38.626164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:38.626398 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:38.626553 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:38.626938 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.627737 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.629317 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.630118 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.630251 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:38.630473 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:38.630527 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 23:53:38.630621 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":106,"threads_started":1}
I20260928 23:53:38.631532 25912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:38.625676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:42553 (local address 127.0.0.1:56322)
0928 23:53:38.626553 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:38.626565 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:38.626692 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:38.627355 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:38.627358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:38.627374 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:38.627459 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:38.627464 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.628751 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.628754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:38.629953 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:38.629962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.631193 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.631196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:38.631282 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:38.631310 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 23:53:38.631379 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 23:53:38.631446 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":161,"thread_start_us":76,"threads_started":1}
I20260928 23:53:38.631546 25910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:38.625262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51594 (local address 127.6.152.2:44683)
0928 23:53:38.626044 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:38.626055 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:38.626080 (+ 25us) server_negotiation.cc:407] Connection header received
0928 23:53:38.626157 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:38.626162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:38.626370 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:38.626886 (+ 516us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:38.627034 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.628026 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.629489 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.630217 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.631077 (+ 860us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:38.631262 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:38.631308 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 23:53:38.631405 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":108,"threads_started":1}
I20260928 23:53:38.631965 25913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:38.625856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56322 (local address 127.6.152.3:42553)
0928 23:53:38.626402 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:38.626428 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:38.626633 (+ 205us) server_negotiation.cc:407] Connection header received
0928 23:53:38.626836 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:38.626841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:38.627071 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:38.627188 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:38.627574 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.628629 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.630071 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.630858 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.630963 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:38.631135 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:38.631726 (+ 591us) server_negotiation.cc:299] Negotiation successful
0928 23:53:38.631817 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":165,"threads_started":1}
I20260928 23:53:38.634030 25569 tablet_service.cc:1467] Processing CreateTablet for tablet 15da86e8674d467499947506f9d3c200 (DEFAULT_TABLE table=test-table [id=7dc60090682e45ae859fa89586f8d5ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:53:38.634029 25700 tablet_service.cc:1467] Processing CreateTablet for tablet 15da86e8674d467499947506f9d3c200 (DEFAULT_TABLE table=test-table [id=7dc60090682e45ae859fa89586f8d5ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:53:38.634227 25831 tablet_service.cc:1467] Processing CreateTablet for tablet 15da86e8674d467499947506f9d3c200 (DEFAULT_TABLE table=test-table [id=7dc60090682e45ae859fa89586f8d5ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:53:38.635062 25831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15da86e8674d467499947506f9d3c200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:38.635073 25569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15da86e8674d467499947506f9d3c200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:38.635073 25700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15da86e8674d467499947506f9d3c200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:38.641731 25914 tablet_bootstrap.cc:492] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16: Bootstrap starting.
I20260928 23:53:38.642299 25915 tablet_bootstrap.cc:492] T 15da86e8674d467499947506f9d3c200 P 3f3595becacc40baab8f3708f1c87a15: Bootstrap starting.
I20260928 23:53:38.642634 25916 tablet_bootstrap.cc:492] T 15da86e8674d467499947506f9d3c200 P 124c0e87339e45cd87337735e6df381a: Bootstrap starting.
I20260928 23:53:38.644104 25914 tablet_bootstrap.cc:654] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:38.644269 25916 tablet_bootstrap.cc:654] T 15da86e8674d467499947506f9d3c200 P 124c0e87339e45cd87337735e6df381a: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:38.644274 25915 tablet_bootstrap.cc:654] T 15da86e8674d467499947506f9d3c200 P 3f3595becacc40baab8f3708f1c87a15: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:38.644976 25916 log.cc:824] T 15da86e8674d467499947506f9d3c200 P 124c0e87339e45cd87337735e6df381a: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:38.645000 25914 log.cc:824] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:38.645000 25915 log.cc:824] T 15da86e8674d467499947506f9d3c200 P 3f3595becacc40baab8f3708f1c87a15: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:38.646764 25914 tablet_bootstrap.cc:492] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16: No bootstrap required, opened a new log
I20260928 23:53:38.646967 25914 ts_tablet_manager.cc:1397] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:53:38.647210 25915 tablet_bootstrap.cc:492] T 15da86e8674d467499947506f9d3c200 P 3f3595becacc40baab8f3708f1c87a15: No bootstrap required, opened a new log
I20260928 23:53:38.647351 25916 tablet_bootstrap.cc:492] T 15da86e8674d467499947506f9d3c200 P 124c0e87339e45cd87337735e6df381a: No bootstrap required, opened a new log
I20260928 23:53:38.647409 25915 ts_tablet_manager.cc:1397] T 15da86e8674d467499947506f9d3c200 P 3f3595becacc40baab8f3708f1c87a15: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 23:53:38.647506 25916 ts_tablet_manager.cc:1397] T 15da86e8674d467499947506f9d3c200 P 124c0e87339e45cd87337735e6df381a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 23:53:38.650230 25914 raft_consensus.cc:348] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124c0e87339e45cd87337735e6df381a" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 37715 } } peers { permanent_uuid: "3f3595becacc40baab8f3708f1c87a15" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 44683 } } peers { permanent_uuid: "a5b43815cd6b4d6d8e141be260a92a16" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 42553 } }
I20260928 23:53:38.650467 25914 raft_consensus.cc:374] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:38.650519 25914 raft_consensus.cc:729] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b43815cd6b4d6d8e141be260a92a16, State: Initialized, Role: FOLLOWER
I20260928 23:53:38.650954 25914 consensus_queue.cc:260] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124c0e87339e45cd87337735e6df381a" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 37715 } } peers { permanent_uuid: "3f3595becacc40baab8f3708f1c87a15" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 44683 } } peers { permanent_uuid: "a5b43815cd6b4d6d8e141be260a92a16" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 42553 } }
I20260928 23:53:38.651335 25916 raft_consensus.cc:348] T 15da86e8674d467499947506f9d3c200 P 124c0e87339e45cd87337735e6df381a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124c0e87339e45cd87337735e6df381a" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 37715 } } peers { permanent_uuid: "3f3595becacc40baab8f3708f1c87a15" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 44683 } } peers { permanent_uuid: "a5b43815cd6b4d6d8e141be260a92a16" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 42553 } }
I20260928 23:53:38.651335 25915 raft_consensus.cc:348] T 15da86e8674d467499947506f9d3c200 P 3f3595becacc40baab8f3708f1c87a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124c0e87339e45cd87337735e6df381a" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 37715 } } peers { permanent_uuid: "3f3595becacc40baab8f3708f1c87a15" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 44683 } } peers { permanent_uuid: "a5b43815cd6b4d6d8e141be260a92a16" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 42553 } }
I20260928 23:53:38.651571 25915 raft_consensus.cc:374] T 15da86e8674d467499947506f9d3c200 P 3f3595becacc40baab8f3708f1c87a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:38.651571 25916 raft_consensus.cc:374] T 15da86e8674d467499947506f9d3c200 P 124c0e87339e45cd87337735e6df381a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:38.651572 25897 heartbeater.cc:503] Master 127.6.152.62:33713 was elected leader, sending a full tablet report...
I20260928 23:53:38.651620 25916 raft_consensus.cc:729] T 15da86e8674d467499947506f9d3c200 P 124c0e87339e45cd87337735e6df381a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124c0e87339e45cd87337735e6df381a, State: Initialized, Role: FOLLOWER
I20260928 23:53:38.651628 25915 raft_consensus.cc:729] T 15da86e8674d467499947506f9d3c200 P 3f3595becacc40baab8f3708f1c87a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3595becacc40baab8f3708f1c87a15, State: Initialized, Role: FOLLOWER
I20260928 23:53:38.651927 25914 ts_tablet_manager.cc:1428] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 23:53:38.652056 25916 consensus_queue.cc:260] T 15da86e8674d467499947506f9d3c200 P 124c0e87339e45cd87337735e6df381a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124c0e87339e45cd87337735e6df381a" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 37715 } } peers { permanent_uuid: "3f3595becacc40baab8f3708f1c87a15" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 44683 } } peers { permanent_uuid: "a5b43815cd6b4d6d8e141be260a92a16" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 42553 } }
I20260928 23:53:38.652056 25915 consensus_queue.cc:260] T 15da86e8674d467499947506f9d3c200 P 3f3595becacc40baab8f3708f1c87a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124c0e87339e45cd87337735e6df381a" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 37715 } } peers { permanent_uuid: "3f3595becacc40baab8f3708f1c87a15" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 44683 } } peers { permanent_uuid: "a5b43815cd6b4d6d8e141be260a92a16" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 42553 } }
I20260928 23:53:38.652714 25766 heartbeater.cc:503] Master 127.6.152.62:33713 was elected leader, sending a full tablet report...
I20260928 23:53:38.652926 25915 ts_tablet_manager.cc:1428] T 15da86e8674d467499947506f9d3c200 P 3f3595becacc40baab8f3708f1c87a15: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 23:53:38.652916 25635 heartbeater.cc:503] Master 127.6.152.62:33713 was elected leader, sending a full tablet report...
I20260928 23:53:38.653437 25916 ts_tablet_manager.cc:1428] T 15da86e8674d467499947506f9d3c200 P 124c0e87339e45cd87337735e6df381a: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
W20260928 23:53:38.656715 25767 tablet.cc:2369] T 15da86e8674d467499947506f9d3c200 P 3f3595becacc40baab8f3708f1c87a15: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:53:38.715945 25636 tablet.cc:2369] T 15da86e8674d467499947506f9d3c200 P 124c0e87339e45cd87337735e6df381a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:53:38.832978 25920 raft_consensus.cc:482] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:53:38.833185 25920 raft_consensus.cc:504] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124c0e87339e45cd87337735e6df381a" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 37715 } } peers { permanent_uuid: "3f3595becacc40baab8f3708f1c87a15" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 44683 } } peers { permanent_uuid: "a5b43815cd6b4d6d8e141be260a92a16" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 42553 } }
I20260928 23:53:38.834277 25920 leader_election.cc:288] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 124c0e87339e45cd87337735e6df381a (127.6.152.1:37715), 3f3595becacc40baab8f3708f1c87a15 (127.6.152.2:44683)
I20260928 23:53:38.837971 25899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:38.834488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:37715 (local address 127.6.152.3:42335)
0928 23:53:38.834606 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:38.834621 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:38.834744 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:38.835052 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:38.835056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:38.835082 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:38.835196 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:38.835204 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.835971 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.835977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:38.836914 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:38.836923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.837544 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.837549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:38.837640 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:38.837668 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 23:53:38.837762 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 23:53:38.837834 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 23:53:38.837971 25911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:38.834629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:42335 (local address 127.6.152.1:37715)
0928 23:53:38.834772 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:38.834779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:38.834797 (+ 18us) server_negotiation.cc:407] Connection header received
0928 23:53:38.834857 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:38.834863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:38.834914 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:38.835003 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:38.835365 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.835851 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.837032 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.837422 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.837559 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:38.837638 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:38.837793 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 23:53:38.837863 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 23:53:38.838578 25589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15da86e8674d467499947506f9d3c200" candidate_uuid: "a5b43815cd6b4d6d8e141be260a92a16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124c0e87339e45cd87337735e6df381a" is_pre_election: true
I20260928 23:53:38.838677 25923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:38.834488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:44683 (local address 127.6.152.3:59649)
0928 23:53:38.835205 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:38.835218 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:38.835318 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:38.835592 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:38.835595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:38.835613 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:38.835735 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:38.835740 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.836439 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.836443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:38.837336 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:38.837345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.838280 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.838284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:38.838377 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:38.838395 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 23:53:38.838479 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 23:53:38.838537 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":330,"thread_start_us":149,"threads_started":1}
I20260928 23:53:38.838672 25910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:38.834638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:59649 (local address 127.6.152.2:44683)
0928 23:53:38.834767 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:38.834770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:38.835304 (+ 534us) server_negotiation.cc:407] Connection header received
0928 23:53:38.835447 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:38.835450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:38.835494 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:38.835571 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:38.835841 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.836315 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.837461 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.837988 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.838129 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:38.838192 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:38.838498 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 23:53:38.838568 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:53:38.838902 25589 raft_consensus.cc:2454] T 15da86e8674d467499947506f9d3c200 P 124c0e87339e45cd87337735e6df381a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5b43815cd6b4d6d8e141be260a92a16 in term 0.
I20260928 23:53:38.839308 25720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15da86e8674d467499947506f9d3c200" candidate_uuid: "a5b43815cd6b4d6d8e141be260a92a16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f3595becacc40baab8f3708f1c87a15" is_pre_election: true
I20260928 23:53:38.839421 25785 leader_election.cc:302] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124c0e87339e45cd87337735e6df381a, a5b43815cd6b4d6d8e141be260a92a16; no voters:
I20260928 23:53:38.839594 25720 raft_consensus.cc:2454] T 15da86e8674d467499947506f9d3c200 P 3f3595becacc40baab8f3708f1c87a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5b43815cd6b4d6d8e141be260a92a16 in term 0.
I20260928 23:53:38.839696 25920 raft_consensus.cc:2781] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:53:38.839807 25920 raft_consensus.cc:482] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:53:38.839861 25920 raft_consensus.cc:3037] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:38.840940 25920 raft_consensus.cc:504] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124c0e87339e45cd87337735e6df381a" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 37715 } } peers { permanent_uuid: "3f3595becacc40baab8f3708f1c87a15" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 44683 } } peers { permanent_uuid: "a5b43815cd6b4d6d8e141be260a92a16" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 42553 } }
I20260928 23:53:38.841267 25920 leader_election.cc:288] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16 [CANDIDATE]: Term 1 election: Requested vote from peers 124c0e87339e45cd87337735e6df381a (127.6.152.1:37715), 3f3595becacc40baab8f3708f1c87a15 (127.6.152.2:44683)
I20260928 23:53:38.841583 25589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15da86e8674d467499947506f9d3c200" candidate_uuid: "a5b43815cd6b4d6d8e141be260a92a16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124c0e87339e45cd87337735e6df381a"
I20260928 23:53:38.841566 25720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15da86e8674d467499947506f9d3c200" candidate_uuid: "a5b43815cd6b4d6d8e141be260a92a16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f3595becacc40baab8f3708f1c87a15"
I20260928 23:53:38.841711 25589 raft_consensus.cc:3037] T 15da86e8674d467499947506f9d3c200 P 124c0e87339e45cd87337735e6df381a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:38.841718 25720 raft_consensus.cc:3037] T 15da86e8674d467499947506f9d3c200 P 3f3595becacc40baab8f3708f1c87a15 [term 0 FOLLOWER]: Advancing to term 1
W20260928 23:53:38.842239 25898 tablet.cc:2369] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:53:38.842772 25589 raft_consensus.cc:2454] T 15da86e8674d467499947506f9d3c200 P 124c0e87339e45cd87337735e6df381a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5b43815cd6b4d6d8e141be260a92a16 in term 1.
I20260928 23:53:38.842986 25720 raft_consensus.cc:2454] T 15da86e8674d467499947506f9d3c200 P 3f3595becacc40baab8f3708f1c87a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5b43815cd6b4d6d8e141be260a92a16 in term 1.
I20260928 23:53:38.843088 25785 leader_election.cc:302] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124c0e87339e45cd87337735e6df381a, a5b43815cd6b4d6d8e141be260a92a16; no voters:
I20260928 23:53:38.843313 25920 raft_consensus.cc:2781] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:53:38.843533 25920 raft_consensus.cc:686] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16 [term 1 LEADER]: Becoming Leader. State: Replica: a5b43815cd6b4d6d8e141be260a92a16, State: Running, Role: LEADER
I20260928 23:53:38.843860 25920 consensus_queue.cc:237] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124c0e87339e45cd87337735e6df381a" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 37715 } } peers { permanent_uuid: "3f3595becacc40baab8f3708f1c87a15" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 44683 } } peers { permanent_uuid: "a5b43815cd6b4d6d8e141be260a92a16" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 42553 } }
I20260928 23:53:38.846903 25447 catalog_manager.cc:5495] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5b43815cd6b4d6d8e141be260a92a16 (127.6.152.3). New cstate: current_term: 1 leader_uuid: "a5b43815cd6b4d6d8e141be260a92a16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124c0e87339e45cd87337735e6df381a" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 37715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f3595becacc40baab8f3708f1c87a15" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 44683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5b43815cd6b4d6d8e141be260a92a16" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 42553 } health_report { overall_health: HEALTHY } } }
I20260928 23:53:38.864629 25913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:38.861589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56326 (local address 127.6.152.3:42553)
0928 23:53:38.861701 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:38.861704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:38.861721 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:53:38.861855 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:38.861857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:38.861916 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:38.861999 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:53:38.862198 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.862665 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.863455 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:38.863649 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:38.863720 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:38.863956 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:53:38.864324 (+ 368us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:53:38.864428 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:38.864507 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 23:53:38.864535 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 23:53:38.870342 25589 raft_consensus.cc:1261] T 15da86e8674d467499947506f9d3c200 P 124c0e87339e45cd87337735e6df381a [term 1 FOLLOWER]: Refusing update from remote peer a5b43815cd6b4d6d8e141be260a92a16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:53:38.870342 25720 raft_consensus.cc:1261] T 15da86e8674d467499947506f9d3c200 P 3f3595becacc40baab8f3708f1c87a15 [term 1 FOLLOWER]: Refusing update from remote peer a5b43815cd6b4d6d8e141be260a92a16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:53:38.871106 25924 consensus_queue.cc:1035] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f3595becacc40baab8f3708f1c87a15" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 44683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:53:38.871307 25920 consensus_queue.cc:1035] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124c0e87339e45cd87337735e6df381a" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 37715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:53:38.878556 25927 mvcc.cc:205] Tried to move back new op lower bound from 7334459878885425152 to 7334459878787088384. Current Snapshot: MvccSnapshot[applied={T|T < 7334459878885425152}]
I20260928 23:53:38.879877 25929 mvcc.cc:205] Tried to move back new op lower bound from 7334459878885425152 to 7334459878787088384. Current Snapshot: MvccSnapshot[applied={T|T < 7334459878885425152}]
I20260928 23:53:38.880028 25928 mvcc.cc:205] Tried to move back new op lower bound from 7334459878885425152 to 7334459878787088384. Current Snapshot: MvccSnapshot[applied={T|T < 7334459878885425152}]
I20260928 23:53:38.888170 25443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:53:38.888345 25443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:53:38.890762 25443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a046260454a0422096b2de7fbd3ecfa6: Sending DeleteTablet for 3 replicas of tablet 15da86e8674d467499947506f9d3c200
I20260928 23:53:38.891486 25569 tablet_service.cc:1514] Processing DeleteTablet for tablet 15da86e8674d467499947506f9d3c200 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:53:38 UTC) from {username='slave'} at 127.0.0.1:54478
I20260928 23:53:38.891705 25700 tablet_service.cc:1514] Processing DeleteTablet for tablet 15da86e8674d467499947506f9d3c200 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:53:38 UTC) from {username='slave'} at 127.0.0.1:51594
I20260928 23:53:38.892074 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 25507
I20260928 23:53:38.892158 25831 tablet_service.cc:1514] Processing DeleteTablet for tablet 15da86e8674d467499947506f9d3c200 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:53:38 UTC) from {username='slave'} at 127.0.0.1:56322
I20260928 23:53:38.892740 25938 tablet_replica.cc:330] T 15da86e8674d467499947506f9d3c200 P 3f3595becacc40baab8f3708f1c87a15: stopping tablet replica
I20260928 23:53:38.892958 25939 tablet_replica.cc:330] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16: stopping tablet replica
I20260928 23:53:38.893263 25939 raft_consensus.cc:2229] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:53:38.893651 25939 raft_consensus.cc:2258] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:53:38.894098 25938 raft_consensus.cc:2229] T 15da86e8674d467499947506f9d3c200 P 3f3595becacc40baab8f3708f1c87a15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:53:38.894388 25938 raft_consensus.cc:2258] T 15da86e8674d467499947506f9d3c200 P 3f3595becacc40baab8f3708f1c87a15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:53:38.895123 25939 ts_tablet_manager.cc:1905] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:53:38.898145 25939 ts_tablet_manager.cc:1918] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:53:38.898279 25939 log.cc:1196] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/15da86e8674d467499947506f9d3c200
I20260928 23:53:38.899446 25939 ts_tablet_manager.cc:1939] T 15da86e8674d467499947506f9d3c200 P a5b43815cd6b4d6d8e141be260a92a16: Deleting consensus metadata
I20260928 23:53:38.900158 25938 ts_tablet_manager.cc:1905] T 15da86e8674d467499947506f9d3c200 P 3f3595becacc40baab8f3708f1c87a15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:53:38.900559 25434 catalog_manager.cc:4841] TS a5b43815cd6b4d6d8e141be260a92a16 (127.6.152.3:42553): tablet 15da86e8674d467499947506f9d3c200 (table test-table [id=7dc60090682e45ae859fa89586f8d5ca]) successfully deleted
W20260928 23:53:38.900662 25432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.152.1:37715 (error 108)
I20260928 23:53:38.901647 25912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:38.901368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:37715 (local address 127.0.0.1:54494)
0928 23:53:38.901499 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:38.901567 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.1:37715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 23:53:38.901726 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 25638
W20260928 23:53:38.905439 25432 catalog_manager.cc:4584] TS 124c0e87339e45cd87337735e6df381a (127.6.152.1:37715): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 15da86e8674d467499947506f9d3c200: Network error: Client connection negotiation failed: client connection to 127.6.152.1:37715: connect: Connection refused (error 111)
I20260928 23:53:38.907979 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 25769
I20260928 23:53:38.909463 25912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:38.908172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:44683 (local address 127.0.0.1:51600)
0928 23:53:38.909293 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:38.909357 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.2:44683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1018}
I20260928 23:53:38.912590 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 25416
2026-09-28T23:53:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1150 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:53:38 dist-test-slave-p5bw krb5kdc[25943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:53:38 dist-test-slave-p5bw krb5kdc[25943](info): set up 2 sockets
Sep 28 23:53:38 dist-test-slave-p5bw krb5kdc[25943](info): commencing operation
krb5kdc: starting...
W20260928 23:53:39.945504 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:53:40 dist-test-slave-p5bw krb5kdc[25943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:53:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:53:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:53:40.064972 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:33475
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:44435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:33475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:53:40.173609 25956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:40.173854 25956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:40.173892 25956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:40.177922 25956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:53:40.177976 25956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:40.177995 25956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:40.178012 25956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:53:40.178059 25956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:53:40.183214 25956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:44435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:33475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:33475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:40.184209 25956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:40.185294 25956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:40.193274 25962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:40.193297 25961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:40.193465 25956 server_base.cc:1034] running on GCE node
W20260928 23:53:40.193269 25964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:40.194221 25956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:40.195281 25956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:40.196462 25956 hybrid_clock.cc:648] HybridClock initialized: now 1790639620196443 us; error 30 us; skew 500 ppm
Sep 28 23:53:40 dist-test-slave-p5bw krb5kdc[25943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639620, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:40.199298 25956 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:53:40.200300 25956 webserver.cc:460] Webserver started at http://127.6.152.62:45773/ using document root <none> and password file <none>
I20260928 23:53:40.200937 25956 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:40.201007 25956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:40.201236 25956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:40.203395 25956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "bc3999ab948d42828abae24f101bbb4e"
format_stamp: "Formatted at 2026-09-28 23:53:40 on dist-test-slave-p5bw"
server_key: "f79acfdb73154d4e4288f7f6226e98a6"
server_key_iv: "3c258bf9b67c10ccdb88c26270991231"
server_key_version: "encryptionkey@0"
I20260928 23:53:40.204017 25956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "bc3999ab948d42828abae24f101bbb4e"
format_stamp: "Formatted at 2026-09-28 23:53:40 on dist-test-slave-p5bw"
server_key: "f79acfdb73154d4e4288f7f6226e98a6"
server_key_iv: "3c258bf9b67c10ccdb88c26270991231"
server_key_version: "encryptionkey@0"
I20260928 23:53:40.207871 25956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 23:53:40.210453 25971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:40.211716 25956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:53:40.211890 25956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "bc3999ab948d42828abae24f101bbb4e"
format_stamp: "Formatted at 2026-09-28 23:53:40 on dist-test-slave-p5bw"
server_key: "f79acfdb73154d4e4288f7f6226e98a6"
server_key_iv: "3c258bf9b67c10ccdb88c26270991231"
server_key_version: "encryptionkey@0"
I20260928 23:53:40.212008 25956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:40.224942 25956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:40.227849 25956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:40.228058 25956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:40.235602 25956 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:33475
I20260928 23:53:40.235666 26023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:33475 every 8 connection(s)
I20260928 23:53:40.236600 25956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:53:40.239327 26024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:40.240682 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 25956
I20260928 23:53:40.240852 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:53:40.241142 6752 external_mini_cluster.cc:1428] Setting key ddb0e5f1593f676468a2dddc0844b28c
I20260928 23:53:40.245813 26024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc3999ab948d42828abae24f101bbb4e: Bootstrap starting.
I20260928 23:53:40.248313 26024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc3999ab948d42828abae24f101bbb4e: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:40.249111 26024 log.cc:824] T 00000000000000000000000000000000 P bc3999ab948d42828abae24f101bbb4e: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:53:40 dist-test-slave-p5bw krb5kdc[25943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:53:40.251144 26024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc3999ab948d42828abae24f101bbb4e: No bootstrap required, opened a new log
I20260928 23:53:40.255831 26024 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc3999ab948d42828abae24f101bbb4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc3999ab948d42828abae24f101bbb4e" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 33475 } }
I20260928 23:53:40.256028 26027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:40.242500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34088 (local address 127.6.152.62:33475)
0928 23:53:40.242918 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:40.242937 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:40.242979 (+ 42us) server_negotiation.cc:407] Connection header received
0928 23:53:40.243809 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:40.243833 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:40.244234 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:40.244594 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:40.245296 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.246686 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.247528 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.248204 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.248379 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:40.251053 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:40.251059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:40.251075 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:53:40.251104 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:40.253042 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:40.253474 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:40.253481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:40.253489 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:40.253658 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:40.253918 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:40.253923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:40.253926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:40.254403 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:40.254606 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:40.254793 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 23:53:40.255099 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":110,"threads_started":1}
I20260928 23:53:40.256084 26024 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc3999ab948d42828abae24f101bbb4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:40.256127 26024 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc3999ab948d42828abae24f101bbb4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc3999ab948d42828abae24f101bbb4e, State: Initialized, Role: FOLLOWER
I20260928 23:53:40.256582 26024 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc3999ab948d42828abae24f101bbb4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc3999ab948d42828abae24f101bbb4e" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 33475 } }
I20260928 23:53:40.256736 26024 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc3999ab948d42828abae24f101bbb4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:53:40.256860 26024 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc3999ab948d42828abae24f101bbb4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:53:40.256965 26024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc3999ab948d42828abae24f101bbb4e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:40.257988 26024 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc3999ab948d42828abae24f101bbb4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc3999ab948d42828abae24f101bbb4e" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 33475 } }
I20260928 23:53:40.258328 26024 leader_election.cc:302] T 00000000000000000000000000000000 P bc3999ab948d42828abae24f101bbb4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc3999ab948d42828abae24f101bbb4e; no voters:
I20260928 23:53:40.258625 26024 leader_election.cc:288] T 00000000000000000000000000000000 P bc3999ab948d42828abae24f101bbb4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:53:40.258710 26029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc3999ab948d42828abae24f101bbb4e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:53:40.258973 26029 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc3999ab948d42828abae24f101bbb4e [term 1 LEADER]: Becoming Leader. State: Replica: bc3999ab948d42828abae24f101bbb4e, State: Running, Role: LEADER
I20260928 23:53:40.259298 26029 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc3999ab948d42828abae24f101bbb4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc3999ab948d42828abae24f101bbb4e" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 33475 } }
I20260928 23:53:40.259773 26024 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc3999ab948d42828abae24f101bbb4e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:53:40.260846 26031 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc3999ab948d42828abae24f101bbb4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc3999ab948d42828abae24f101bbb4e. Latest consensus state: current_term: 1 leader_uuid: "bc3999ab948d42828abae24f101bbb4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc3999ab948d42828abae24f101bbb4e" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 33475 } } }
I20260928 23:53:40.260977 26031 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc3999ab948d42828abae24f101bbb4e [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:40.261353 26038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:53:40.261315 26030 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc3999ab948d42828abae24f101bbb4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc3999ab948d42828abae24f101bbb4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc3999ab948d42828abae24f101bbb4e" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 33475 } } }
I20260928 23:53:40.262254 26030 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc3999ab948d42828abae24f101bbb4e [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:40.264295 26038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:53:40.269747 26038 catalog_manager.cc:1347] Generated new cluster ID: 700005fa3d424ec78eb516a7708a28e6
I20260928 23:53:40.269809 26038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:53:40.278748 26038 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:53:40.279886 26038 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:53:40.287503 26038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc3999ab948d42828abae24f101bbb4e: Generated new TSK 0
I20260928 23:53:40.288322 26038 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
Entry for principal kudu/127.6.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab.
I20260928 23:53:40.317947 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:33475
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:44435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:53:40.425662 26050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:40.425904 26050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:40.425977 26050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:40.429980 26050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:40.430037 26050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:40.430128 26050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:53:40.435441 26050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:44435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.1.keytab
--principal=kudu/127.6.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:33475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:40.436470 26050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:40.437484 26050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:40.446892 26058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:40.446950 26056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:40.447042 26050 server_base.cc:1034] running on GCE node
W20260928 23:53:40.447077 26055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:40.447479 26050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:40.448066 26050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:40.449219 26050 hybrid_clock.cc:648] HybridClock initialized: now 1790639620449205 us; error 25 us; skew 500 ppm
Sep 28 23:53:40 dist-test-slave-p5bw krb5kdc[25943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639620, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:40.451892 26050 init.cc:377] Logged in from keytab as kudu/127.6.152.1@KRBTEST.COM (short username kudu)
I20260928 23:53:40.452713 26050 webserver.cc:460] Webserver started at http://127.6.152.1:34129/ using document root <none> and password file <none>
I20260928 23:53:40.453297 26050 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:40.453357 26050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:40.453516 26050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:40.455415 26050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "347e4289f36f4aaa8fbfb0c17e79c8d1"
format_stamp: "Formatted at 2026-09-28 23:53:40 on dist-test-slave-p5bw"
server_key: "528876685366014f03c5d6c25393d3a4"
server_key_iv: "a8030a4922492860751d0e9c04105555"
server_key_version: "encryptionkey@0"
I20260928 23:53:40.455888 26050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "347e4289f36f4aaa8fbfb0c17e79c8d1"
format_stamp: "Formatted at 2026-09-28 23:53:40 on dist-test-slave-p5bw"
server_key: "528876685366014f03c5d6c25393d3a4"
server_key_iv: "a8030a4922492860751d0e9c04105555"
server_key_version: "encryptionkey@0"
I20260928 23:53:40.459403 26050 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 23:53:40.461954 26065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:40.463122 26050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:53:40.463272 26050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "347e4289f36f4aaa8fbfb0c17e79c8d1"
format_stamp: "Formatted at 2026-09-28 23:53:40 on dist-test-slave-p5bw"
server_key: "528876685366014f03c5d6c25393d3a4"
server_key_iv: "a8030a4922492860751d0e9c04105555"
server_key_version: "encryptionkey@0"
I20260928 23:53:40.463382 26050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:40.475111 26050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:40.478269 26050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:40.478469 26050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:40.479027 26050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:53:40.479866 26050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:53:40.479941 26050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:40.480014 26050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:53:40.480067 26050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:40.488945 26050 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:46269
I20260928 23:53:40.488967 26178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:46269 every 8 connection(s)
I20260928 23:53:40.489838 26050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:53:40.493765 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 26050
I20260928 23:53:40.493921 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
I20260928 23:53:40.494220 6752 external_mini_cluster.cc:1428] Setting key 78a25c42794c2b6529effce879b9f98e
Sep 28 23:53:40 dist-test-slave-p5bw krb5kdc[25943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639620, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.1@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.152.2@KRBTEST.COM; defaulting to no policy
I20260928 23:53:40.503450 26027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:40.491543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:48477 (local address 127.6.152.62:33475)
0928 23:53:40.491683 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:40.491687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:40.492743 (+ 1056us) server_negotiation.cc:407] Connection header received
0928 23:53:40.493648 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:40.493652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:40.493724 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:40.493826 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:40.495069 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.495593 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.496742 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.497035 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.497145 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:40.499587 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:40.499593 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:40.499599 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:53:40.499632 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:40.501202 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:40.501829 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:40.501831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:40.501832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:40.501903 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:40.502291 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:40.502293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:40.502294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:40.502459 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:40.502532 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:40.503177 (+ 645us) server_negotiation.cc:299] Negotiation successful
0928 23:53:40.503287 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:53:40.504388 26181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:40.491900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:33475 (local address 127.6.152.1:48477)
0928 23:53:40.492584 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:40.492626 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:40.493429 (+ 803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:40.493995 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:40.494024 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:40.494452 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:53:40.494918 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:40.494931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.495717 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.495723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:40.496610 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:40.496620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.497192 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.497197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:40.497362 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:40.497957 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:53:40.497973 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:53:40.499424 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:53:40.501319 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:53:40.501326 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:53:40.501339 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:53:40.501722 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:53:40.502006 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:53:40.502009 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:53:40.502012 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:53:40.502191 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:53:40.502569 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:53:40.502577 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:53:40.502926 (+ 349us) client_negotiation.cc:769] Sending connection context
0928 23:53:40.503143 (+ 217us) client_negotiation.cc:240] Negotiation successful
0928 23:53:40.503506 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":117,"threads_started":1}
I20260928 23:53:40.505631 26179 heartbeater.cc:347] Connected to a master server at 127.6.152.62:33475
I20260928 23:53:40.505872 26179 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.2@KRBTEST.COM" created.
I20260928 23:53:40.506438 26179 heartbeater.cc:511] Master 127.6.152.62:33475 requested a full tablet report, sending...
I20260928 23:53:40.508045 25988 ts_manager.cc:194] Registered new tserver with Master: 347e4289f36f4aaa8fbfb0c17e79c8d1 (127.6.152.1:46269)
I20260928 23:53:40.509215 25988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.1@KRBTEST.COM'} at 127.6.152.1:48477
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
Entry for principal kudu/127.6.152.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab.
I20260928 23:53:40.521802 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:33475
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:44435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:53:40.628328 26184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:40.628561 26184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:40.628618 26184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:40.632563 26184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:40.632617 26184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:40.632711 26184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:53:40.637822 26184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:44435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.2.keytab
--principal=kudu/127.6.152.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:33475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:40.638779 26184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:40.639734 26184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:53:40.648931 26184 server_base.cc:1034] running on GCE node
W20260928 23:53:40.648962 26192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:40.648936 26190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:40.648962 26189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:40.649524 26184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:40.650101 26184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:40.651279 26184 hybrid_clock.cc:648] HybridClock initialized: now 1790639620651270 us; error 28 us; skew 500 ppm
Sep 28 23:53:40 dist-test-slave-p5bw krb5kdc[25943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639620, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:40.653905 26184 init.cc:377] Logged in from keytab as kudu/127.6.152.2@KRBTEST.COM (short username kudu)
I20260928 23:53:40.654671 26184 webserver.cc:460] Webserver started at http://127.6.152.2:33931/ using document root <none> and password file <none>
I20260928 23:53:40.655190 26184 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:40.655275 26184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:40.655503 26184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:40.657473 26184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "60e4953c2af2441d9ea676247f9e2d07"
format_stamp: "Formatted at 2026-09-28 23:53:40 on dist-test-slave-p5bw"
server_key: "507fa19ff602b5e92fe40e8c573dd4e4"
server_key_iv: "9120c0effdaa673eae3a188f34ba6059"
server_key_version: "encryptionkey@0"
I20260928 23:53:40.657996 26184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "60e4953c2af2441d9ea676247f9e2d07"
format_stamp: "Formatted at 2026-09-28 23:53:40 on dist-test-slave-p5bw"
server_key: "507fa19ff602b5e92fe40e8c573dd4e4"
server_key_iv: "9120c0effdaa673eae3a188f34ba6059"
server_key_version: "encryptionkey@0"
I20260928 23:53:40.661710 26184 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:53:40.663903 26199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:40.665100 26184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:53:40.665221 26184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "60e4953c2af2441d9ea676247f9e2d07"
format_stamp: "Formatted at 2026-09-28 23:53:40 on dist-test-slave-p5bw"
server_key: "507fa19ff602b5e92fe40e8c573dd4e4"
server_key_iv: "9120c0effdaa673eae3a188f34ba6059"
server_key_version: "encryptionkey@0"
I20260928 23:53:40.665324 26184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:40.678653 26184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:40.681599 26184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:40.681784 26184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:40.682312 26184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:53:40.683101 26184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:53:40.683169 26184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:40.683252 26184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:53:40.683295 26184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:40.691344 26184 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:33271
I20260928 23:53:40.691411 26312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:33271 every 8 connection(s)
I20260928 23:53:40.692201 26184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:53:40.697553 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 26184
I20260928 23:53:40.697722 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:53:40.698035 6752 external_mini_cluster.cc:1428] Setting key 7a558bb5dc289fc305ce24a67d17fece
Sep 28 23:53:40 dist-test-slave-p5bw krb5kdc[25943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639620, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.2@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:53:40.705195 26027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:40.693910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:41983 (local address 127.6.152.62:33475)
0928 23:53:40.694037 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:40.694041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:40.694819 (+ 778us) server_negotiation.cc:407] Connection header received
0928 23:53:40.695699 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:40.695703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:40.695761 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:40.695833 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:40.697003 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.697535 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.698858 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.699094 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.699205 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:40.701616 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:40.701620 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:40.701623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:53:40.701653 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:40.703157 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:40.703677 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:40.703682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:40.703684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:40.703764 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:40.704098 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:40.704102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:40.704104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:40.704282 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:40.704388 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:40.704904 (+ 516us) server_negotiation.cc:299] Negotiation successful
0928 23:53:40.705028 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:53:40.705988 26315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:40.694249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:33475 (local address 127.6.152.2:41983)
0928 23:53:40.694683 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:40.694722 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:40.695494 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:40.695999 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:40.696006 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:40.696333 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:53:40.696869 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:40.696885 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.697662 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.697666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:40.698735 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:40.698747 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.699240 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.699245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:40.699413 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:40.699989 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:53:40.700005 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:53:40.701459 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:53:40.703282 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:53:40.703288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:53:40.703302 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:53:40.703570 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:53:40.703855 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:53:40.703859 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:53:40.703861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:53:40.704000 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:53:40.704385 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:53:40.704392 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:53:40.704645 (+ 253us) client_negotiation.cc:769] Sending connection context
0928 23:53:40.704846 (+ 201us) client_negotiation.cc:240] Negotiation successful
0928 23:53:40.705170 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.6.152.3@KRBTEST.COM; defaulting to no policy
I20260928 23:53:40.707108 26313 heartbeater.cc:347] Connected to a master server at 127.6.152.62:33475
I20260928 23:53:40.707307 26313 heartbeater.cc:464] Registering TS with master...
I20260928 23:53:40.707800 26313 heartbeater.cc:511] Master 127.6.152.62:33475 requested a full tablet report, sending...
I20260928 23:53:40.708863 25988 ts_manager.cc:194] Registered new tserver with Master: 60e4953c2af2441d9ea676247f9e2d07 (127.6.152.2:33271)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.3@KRBTEST.COM" created.
I20260928 23:53:40.709540 25988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.2@KRBTEST.COM'} at 127.6.152.2:41983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
Entry for principal kudu/127.6.152.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab.
I20260928 23:53:40.724283 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:33475
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:44435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:53:40.830942 26318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:40.831180 26318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:40.831238 26318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:40.835296 26318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:40.835350 26318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:40.835413 26318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:53:40.840454 26318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:44435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.3.keytab
--principal=kudu/127.6.152.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:33475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:40.841392 26318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:40.842272 26318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:40.851636 26326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:40.851599 26323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:40.851701 26318 server_base.cc:1034] running on GCE node
W20260928 23:53:40.851622 26324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:40.852330 26318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:40.852967 26318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:40.854140 26318 hybrid_clock.cc:648] HybridClock initialized: now 1790639620854131 us; error 31 us; skew 500 ppm
Sep 28 23:53:40 dist-test-slave-p5bw krb5kdc[25943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639620, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:40.856693 26318 init.cc:377] Logged in from keytab as kudu/127.6.152.3@KRBTEST.COM (short username kudu)
I20260928 23:53:40.857527 26318 webserver.cc:460] Webserver started at http://127.6.152.3:39195/ using document root <none> and password file <none>
I20260928 23:53:40.858064 26318 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:40.858150 26318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:40.858386 26318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:40.860412 26318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "44abb8018a2b4cffa0fdfd882398b71f"
format_stamp: "Formatted at 2026-09-28 23:53:40 on dist-test-slave-p5bw"
server_key: "8455ce9d7a2edf87b5a0230e5a8d733d"
server_key_iv: "df2a4a4cb4e5c829d7d3bc6ca7290e94"
server_key_version: "encryptionkey@0"
I20260928 23:53:40.861045 26318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "44abb8018a2b4cffa0fdfd882398b71f"
format_stamp: "Formatted at 2026-09-28 23:53:40 on dist-test-slave-p5bw"
server_key: "8455ce9d7a2edf87b5a0230e5a8d733d"
server_key_iv: "df2a4a4cb4e5c829d7d3bc6ca7290e94"
server_key_version: "encryptionkey@0"
I20260928 23:53:40.864446 26318 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:53:40.866812 26333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:40.868005 26318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:53:40.868131 26318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "44abb8018a2b4cffa0fdfd882398b71f"
format_stamp: "Formatted at 2026-09-28 23:53:40 on dist-test-slave-p5bw"
server_key: "8455ce9d7a2edf87b5a0230e5a8d733d"
server_key_iv: "df2a4a4cb4e5c829d7d3bc6ca7290e94"
server_key_version: "encryptionkey@0"
I20260928 23:53:40.868235 26318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:40.895392 26318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:40.898365 26318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:40.898567 26318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:40.899148 26318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:53:40.900012 26318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:53:40.900077 26318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:40.900143 26318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:53:40.900193 26318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:40.909201 26318 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:37821
I20260928 23:53:40.909219 26446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:37821 every 8 connection(s)
I20260928 23:53:40.910167 26318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
Sep 28 23:53:40 dist-test-slave-p5bw krb5kdc[25943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639620, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.3@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:53:40.920409 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 26318
I20260928 23:53:40.920550 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:53:40.920931 6752 external_mini_cluster.cc:1428] Setting key ae7fe4b75004f5ad9f8a092470a75917
I20260928 23:53:40.923055 26027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:40.911928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:54663 (local address 127.6.152.62:33475)
0928 23:53:40.912065 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:40.912069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:40.912986 (+ 917us) server_negotiation.cc:407] Connection header received
0928 23:53:40.913831 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:40.913833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:40.913878 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:40.913953 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:40.915097 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.915694 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.916931 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.917157 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.917234 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:40.919504 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:40.919507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:40.919509 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:53:40.919531 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:40.920959 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:40.921527 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:40.921531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:40.921533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:40.921618 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:40.922010 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:40.922014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:40.922015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:40.922207 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:40.922317 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:40.922772 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 23:53:40.922892 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:53:40.923838 26449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:40.912322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:33475 (local address 127.6.152.3:54663)
0928 23:53:40.912773 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:40.912861 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:40.913642 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:40.914113 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:40.914120 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:40.914474 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:53:40.914957 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:40.914970 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.915809 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.915812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:40.916783 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:40.916810 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.917277 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.917279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:40.917420 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:40.918006 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:53:40.918022 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:53:40.919352 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:53:40.921092 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:53:40.921098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:53:40.921111 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:53:40.921409 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:53:40.921729 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:53:40.921735 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:53:40.921740 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:53:40.921917 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:53:40.922322 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:53:40.922330 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:53:40.922547 (+ 217us) client_negotiation.cc:769] Sending connection context
0928 23:53:40.922730 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 23:53:40.923021 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":111,"threads_started":1}
I20260928 23:53:40.924862 26447 heartbeater.cc:347] Connected to a master server at 127.6.152.62:33475
I20260928 23:53:40.925086 26447 heartbeater.cc:464] Registering TS with master...
I20260928 23:53:40.925560 26447 heartbeater.cc:511] Master 127.6.152.62:33475 requested a full tablet report, sending...
I20260928 23:53:40.926515 25988 ts_manager.cc:194] Registered new tserver with Master: 44abb8018a2b4cffa0fdfd882398b71f (127.6.152.3:37821)
I20260928 23:53:40.927063 25988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.152.3@KRBTEST.COM'} at 127.6.152.3:54663
I20260928 23:53:40.934628 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:53:40.943396 26027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:40.936455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34104 (local address 127.6.152.62:33475)
0928 23:53:40.936600 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:40.936605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:40.936741 (+ 136us) server_negotiation.cc:407] Connection header received
0928 23:53:40.936905 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:40.936907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:40.936947 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:40.937010 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:40.937608 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.938100 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.938863 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.939070 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.939142 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:40.940461 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:40.940464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:40.940466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:53:40.940487 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:40.941788 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:40.942194 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:40.942197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:40.942198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:40.942280 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:40.942551 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:40.942554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:40.942555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:40.942786 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:40.942888 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:40.943101 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 23:53:40.943245 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 23:53:40.946187 25988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:53:40.949553 25988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:53:40.964197 26459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:40.958009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:33271 (local address 127.0.0.1:41076)
0928 23:53:40.958864 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:40.958914 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:40.959504 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:40.959951 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:40.959954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:40.959973 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:40.960085 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:40.960094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.961786 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.961790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:40.962682 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:40.962692 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.963651 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.963655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:40.963760 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:40.963803 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 23:53:40.963918 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 23:53:40.964040 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":16,"thread_start_us":87,"threads_started":1}
I20260928 23:53:40.964219 26460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:40.958133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41076 (local address 127.6.152.2:33271)
0928 23:53:40.958560 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:40.958571 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:40.959473 (+ 902us) server_negotiation.cc:407] Connection header received
0928 23:53:40.959657 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:40.959662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:40.959807 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:40.959962 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:40.960404 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.961630 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.962808 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.963535 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.963641 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:40.963803 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:40.963959 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 23:53:40.964057 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":120,"threads_started":1}
I20260928 23:53:40.964695 26462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:40.958764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:37821 (local address 127.0.0.1:33310)
0928 23:53:40.959281 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:40.959294 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:40.959389 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:40.960398 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:40.960401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:40.960429 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:40.960513 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:40.960519 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.962148 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.962152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:40.963020 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:40.963030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.964286 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.964292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:40.964406 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:40.964428 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:53:40.964499 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 23:53:40.964562 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":87,"threads_started":1}
I20260928 23:53:40.965095 26458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:40.958031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:46269 (local address 127.0.0.1:50216)
0928 23:53:40.958896 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:40.958932 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:40.959061 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:40.959841 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:40.959847 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:40.959870 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:40.960008 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:40.960015 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.961380 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.961386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:40.963402 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:40.963421 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.964654 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.964658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:40.964746 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:40.964771 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 23:53:40.964902 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 23:53:40.964975 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":25,"thread_start_us":118,"threads_started":1}
I20260928 23:53:40.965418 26463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:40.959166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33310 (local address 127.6.152.3:37821)
0928 23:53:40.959755 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:40.959769 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:40.959792 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:53:40.959862 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:40.959867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:40.960042 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:40.960261 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:40.960639 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.961976 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.963154 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.964004 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.964099 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:40.964221 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:40.964537 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 23:53:40.965268 (+ 731us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":130,"threads_started":1}
I20260928 23:53:40.965615 26461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:40.958423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50216 (local address 127.6.152.1:46269)
0928 23:53:40.958782 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:40.958793 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:40.959007 (+ 214us) server_negotiation.cc:407] Connection header received
0928 23:53:40.959243 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:40.959250 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:40.959472 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:40.959611 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:40.960187 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.961255 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.963540 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:40.964361 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:40.964463 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:40.964592 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:40.965350 (+ 758us) server_negotiation.cc:299] Negotiation successful
0928 23:53:40.965448 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":83,"threads_started":1}
I20260928 23:53:40.968154 26247 tablet_service.cc:1467] Processing CreateTablet for tablet 42c94d3421fe48a9a371241c3eac9448 (DEFAULT_TABLE table=test-table [id=a4ed8c58593c42378e896c4aa1327f29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:53:40.968154 26381 tablet_service.cc:1467] Processing CreateTablet for tablet 42c94d3421fe48a9a371241c3eac9448 (DEFAULT_TABLE table=test-table [id=a4ed8c58593c42378e896c4aa1327f29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:53:40.969323 26381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42c94d3421fe48a9a371241c3eac9448. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:40.969323 26247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42c94d3421fe48a9a371241c3eac9448. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:40.975303 26113 tablet_service.cc:1467] Processing CreateTablet for tablet 42c94d3421fe48a9a371241c3eac9448 (DEFAULT_TABLE table=test-table [id=a4ed8c58593c42378e896c4aa1327f29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:53:40.976259 26113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42c94d3421fe48a9a371241c3eac9448. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:40.976964 26464 tablet_bootstrap.cc:492] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07: Bootstrap starting.
I20260928 23:53:40.977716 26465 tablet_bootstrap.cc:492] T 42c94d3421fe48a9a371241c3eac9448 P 44abb8018a2b4cffa0fdfd882398b71f: Bootstrap starting.
I20260928 23:53:40.979126 26464 tablet_bootstrap.cc:654] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:40.979866 26465 tablet_bootstrap.cc:654] T 42c94d3421fe48a9a371241c3eac9448 P 44abb8018a2b4cffa0fdfd882398b71f: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:40.980232 26464 log.cc:824] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:40.981117 26465 log.cc:824] T 42c94d3421fe48a9a371241c3eac9448 P 44abb8018a2b4cffa0fdfd882398b71f: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:40.982337 26464 tablet_bootstrap.cc:492] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07: No bootstrap required, opened a new log
I20260928 23:53:40.982545 26464 ts_tablet_manager.cc:1397] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:53:40.983101 26465 tablet_bootstrap.cc:492] T 42c94d3421fe48a9a371241c3eac9448 P 44abb8018a2b4cffa0fdfd882398b71f: No bootstrap required, opened a new log
I20260928 23:53:40.983363 26465 ts_tablet_manager.cc:1397] T 42c94d3421fe48a9a371241c3eac9448 P 44abb8018a2b4cffa0fdfd882398b71f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:53:40.984340 26468 tablet_bootstrap.cc:492] T 42c94d3421fe48a9a371241c3eac9448 P 347e4289f36f4aaa8fbfb0c17e79c8d1: Bootstrap starting.
I20260928 23:53:40.986532 26468 tablet_bootstrap.cc:654] T 42c94d3421fe48a9a371241c3eac9448 P 347e4289f36f4aaa8fbfb0c17e79c8d1: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:40.986472 26465 raft_consensus.cc:348] T 42c94d3421fe48a9a371241c3eac9448 P 44abb8018a2b4cffa0fdfd882398b71f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e4953c2af2441d9ea676247f9e2d07" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 33271 } } peers { permanent_uuid: "347e4289f36f4aaa8fbfb0c17e79c8d1" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 46269 } } peers { permanent_uuid: "44abb8018a2b4cffa0fdfd882398b71f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 37821 } }
I20260928 23:53:40.986764 26465 raft_consensus.cc:374] T 42c94d3421fe48a9a371241c3eac9448 P 44abb8018a2b4cffa0fdfd882398b71f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:40.986583 26464 raft_consensus.cc:348] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e4953c2af2441d9ea676247f9e2d07" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 33271 } } peers { permanent_uuid: "347e4289f36f4aaa8fbfb0c17e79c8d1" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 46269 } } peers { permanent_uuid: "44abb8018a2b4cffa0fdfd882398b71f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 37821 } }
I20260928 23:53:40.986836 26465 raft_consensus.cc:729] T 42c94d3421fe48a9a371241c3eac9448 P 44abb8018a2b4cffa0fdfd882398b71f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44abb8018a2b4cffa0fdfd882398b71f, State: Initialized, Role: FOLLOWER
I20260928 23:53:40.986891 26464 raft_consensus.cc:374] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:40.986967 26464 raft_consensus.cc:729] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e4953c2af2441d9ea676247f9e2d07, State: Initialized, Role: FOLLOWER
I20260928 23:53:40.987330 26465 consensus_queue.cc:260] T 42c94d3421fe48a9a371241c3eac9448 P 44abb8018a2b4cffa0fdfd882398b71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e4953c2af2441d9ea676247f9e2d07" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 33271 } } peers { permanent_uuid: "347e4289f36f4aaa8fbfb0c17e79c8d1" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 46269 } } peers { permanent_uuid: "44abb8018a2b4cffa0fdfd882398b71f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 37821 } }
I20260928 23:53:40.987440 26464 consensus_queue.cc:260] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e4953c2af2441d9ea676247f9e2d07" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 33271 } } peers { permanent_uuid: "347e4289f36f4aaa8fbfb0c17e79c8d1" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 46269 } } peers { permanent_uuid: "44abb8018a2b4cffa0fdfd882398b71f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 37821 } }
I20260928 23:53:40.987773 26468 log.cc:824] T 42c94d3421fe48a9a371241c3eac9448 P 347e4289f36f4aaa8fbfb0c17e79c8d1: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:40.987941 26447 heartbeater.cc:503] Master 127.6.152.62:33475 was elected leader, sending a full tablet report...
I20260928 23:53:40.988379 26464 ts_tablet_manager.cc:1428] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:53:40.988736 26313 heartbeater.cc:503] Master 127.6.152.62:33475 was elected leader, sending a full tablet report...
I20260928 23:53:40.989516 26465 ts_tablet_manager.cc:1428] T 42c94d3421fe48a9a371241c3eac9448 P 44abb8018a2b4cffa0fdfd882398b71f: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:53:40.991165 26468 tablet_bootstrap.cc:492] T 42c94d3421fe48a9a371241c3eac9448 P 347e4289f36f4aaa8fbfb0c17e79c8d1: No bootstrap required, opened a new log
I20260928 23:53:40.991356 26468 ts_tablet_manager.cc:1397] T 42c94d3421fe48a9a371241c3eac9448 P 347e4289f36f4aaa8fbfb0c17e79c8d1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 23:53:40.994582 26468 raft_consensus.cc:348] T 42c94d3421fe48a9a371241c3eac9448 P 347e4289f36f4aaa8fbfb0c17e79c8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e4953c2af2441d9ea676247f9e2d07" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 33271 } } peers { permanent_uuid: "347e4289f36f4aaa8fbfb0c17e79c8d1" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 46269 } } peers { permanent_uuid: "44abb8018a2b4cffa0fdfd882398b71f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 37821 } }
I20260928 23:53:40.994803 26468 raft_consensus.cc:374] T 42c94d3421fe48a9a371241c3eac9448 P 347e4289f36f4aaa8fbfb0c17e79c8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:40.994858 26468 raft_consensus.cc:729] T 42c94d3421fe48a9a371241c3eac9448 P 347e4289f36f4aaa8fbfb0c17e79c8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 347e4289f36f4aaa8fbfb0c17e79c8d1, State: Initialized, Role: FOLLOWER
I20260928 23:53:40.995260 26468 consensus_queue.cc:260] T 42c94d3421fe48a9a371241c3eac9448 P 347e4289f36f4aaa8fbfb0c17e79c8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e4953c2af2441d9ea676247f9e2d07" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 33271 } } peers { permanent_uuid: "347e4289f36f4aaa8fbfb0c17e79c8d1" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 46269 } } peers { permanent_uuid: "44abb8018a2b4cffa0fdfd882398b71f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 37821 } }
I20260928 23:53:40.995864 26179 heartbeater.cc:503] Master 127.6.152.62:33475 was elected leader, sending a full tablet report...
I20260928 23:53:40.996121 26468 ts_tablet_manager.cc:1428] T 42c94d3421fe48a9a371241c3eac9448 P 347e4289f36f4aaa8fbfb0c17e79c8d1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 23:53:41.032941 26470 raft_consensus.cc:482] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:53:41.033167 26470 raft_consensus.cc:504] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e4953c2af2441d9ea676247f9e2d07" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 33271 } } peers { permanent_uuid: "347e4289f36f4aaa8fbfb0c17e79c8d1" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 46269 } } peers { permanent_uuid: "44abb8018a2b4cffa0fdfd882398b71f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 37821 } }
I20260928 23:53:41.034348 26470 leader_election.cc:288] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 347e4289f36f4aaa8fbfb0c17e79c8d1 (127.6.152.1:46269), 44abb8018a2b4cffa0fdfd882398b71f (127.6.152.3:37821)
I20260928 23:53:41.038300 26463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:41.034748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:41295 (local address 127.6.152.3:37821)
0928 23:53:41.034899 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:41.034913 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:41.034931 (+ 18us) server_negotiation.cc:407] Connection header received
0928 23:53:41.034994 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:41.034997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:41.035036 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:41.035118 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:41.035419 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.035946 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.037162 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.037697 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.037797 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:41.037867 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:41.038143 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 23:53:41.038204 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:53:41.038331 26315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:41.034603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:37821 (local address 127.6.152.2:41295)
0928 23:53:41.034737 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:41.034754 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:41.034876 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:41.035128 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:41.035131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:41.035148 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:41.035298 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:41.035305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.036076 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.036079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:41.036983 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:41.036992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.037909 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.037912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:41.038018 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:41.038042 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 23:53:41.038117 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 23:53:41.038198 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 23:53:41.038934 26473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:41.034616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:46269 (local address 127.6.152.2:45883)
0928 23:53:41.035210 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:41.035222 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:41.035366 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:41.035640 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:41.035643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:41.035658 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:41.035774 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:41.035780 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.036563 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.036568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:41.037596 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:41.037605 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.038541 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.038544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:41.038628 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:41.038656 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 23:53:41.038743 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 23:53:41.038815 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":217,"thread_start_us":133,"threads_started":1}
I20260928 23:53:41.038976 26461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:41.034680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:45883 (local address 127.6.152.1:46269)
0928 23:53:41.034810 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:41.034815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:41.035334 (+ 519us) server_negotiation.cc:407] Connection header received
0928 23:53:41.035490 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:41.035493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:41.035543 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:41.035618 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:41.035906 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.036446 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.037724 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.038299 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.038406 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:41.038469 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:41.038782 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 23:53:41.038865 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:53:41.039103 26401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42c94d3421fe48a9a371241c3eac9448" candidate_uuid: "60e4953c2af2441d9ea676247f9e2d07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44abb8018a2b4cffa0fdfd882398b71f" is_pre_election: true
I20260928 23:53:41.039441 26401 raft_consensus.cc:2454] T 42c94d3421fe48a9a371241c3eac9448 P 44abb8018a2b4cffa0fdfd882398b71f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60e4953c2af2441d9ea676247f9e2d07 in term 0.
I20260928 23:53:41.039579 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42c94d3421fe48a9a371241c3eac9448" candidate_uuid: "60e4953c2af2441d9ea676247f9e2d07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "347e4289f36f4aaa8fbfb0c17e79c8d1" is_pre_election: true
I20260928 23:53:41.039848 26133 raft_consensus.cc:2454] T 42c94d3421fe48a9a371241c3eac9448 P 347e4289f36f4aaa8fbfb0c17e79c8d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60e4953c2af2441d9ea676247f9e2d07 in term 0.
I20260928 23:53:41.039950 26203 leader_election.cc:302] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44abb8018a2b4cffa0fdfd882398b71f, 60e4953c2af2441d9ea676247f9e2d07; no voters:
I20260928 23:53:41.040266 26470 raft_consensus.cc:2781] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:53:41.040347 26470 raft_consensus.cc:482] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:53:41.040377 26470 raft_consensus.cc:3037] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:41.041441 26470 raft_consensus.cc:504] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e4953c2af2441d9ea676247f9e2d07" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 33271 } } peers { permanent_uuid: "347e4289f36f4aaa8fbfb0c17e79c8d1" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 46269 } } peers { permanent_uuid: "44abb8018a2b4cffa0fdfd882398b71f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 37821 } }
I20260928 23:53:41.041810 26470 leader_election.cc:288] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07 [CANDIDATE]: Term 1 election: Requested vote from peers 347e4289f36f4aaa8fbfb0c17e79c8d1 (127.6.152.1:46269), 44abb8018a2b4cffa0fdfd882398b71f (127.6.152.3:37821)
I20260928 23:53:41.042119 26401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42c94d3421fe48a9a371241c3eac9448" candidate_uuid: "60e4953c2af2441d9ea676247f9e2d07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44abb8018a2b4cffa0fdfd882398b71f"
I20260928 23:53:41.042116 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42c94d3421fe48a9a371241c3eac9448" candidate_uuid: "60e4953c2af2441d9ea676247f9e2d07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "347e4289f36f4aaa8fbfb0c17e79c8d1"
I20260928 23:53:41.042236 26133 raft_consensus.cc:3037] T 42c94d3421fe48a9a371241c3eac9448 P 347e4289f36f4aaa8fbfb0c17e79c8d1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:41.042236 26401 raft_consensus.cc:3037] T 42c94d3421fe48a9a371241c3eac9448 P 44abb8018a2b4cffa0fdfd882398b71f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:41.043489 26401 raft_consensus.cc:2454] T 42c94d3421fe48a9a371241c3eac9448 P 44abb8018a2b4cffa0fdfd882398b71f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60e4953c2af2441d9ea676247f9e2d07 in term 1.
I20260928 23:53:41.043493 26133 raft_consensus.cc:2454] T 42c94d3421fe48a9a371241c3eac9448 P 347e4289f36f4aaa8fbfb0c17e79c8d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60e4953c2af2441d9ea676247f9e2d07 in term 1.
I20260928 23:53:41.043772 26203 leader_election.cc:302] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44abb8018a2b4cffa0fdfd882398b71f, 60e4953c2af2441d9ea676247f9e2d07; no voters:
I20260928 23:53:41.043956 26470 raft_consensus.cc:2781] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:53:41.044180 26470 raft_consensus.cc:686] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07 [term 1 LEADER]: Becoming Leader. State: Replica: 60e4953c2af2441d9ea676247f9e2d07, State: Running, Role: LEADER
I20260928 23:53:41.044507 26470 consensus_queue.cc:237] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e4953c2af2441d9ea676247f9e2d07" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 33271 } } peers { permanent_uuid: "347e4289f36f4aaa8fbfb0c17e79c8d1" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 46269 } } peers { permanent_uuid: "44abb8018a2b4cffa0fdfd882398b71f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 37821 } }
I20260928 23:53:41.047700 25988 catalog_manager.cc:5495] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60e4953c2af2441d9ea676247f9e2d07 (127.6.152.2). New cstate: current_term: 1 leader_uuid: "60e4953c2af2441d9ea676247f9e2d07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e4953c2af2441d9ea676247f9e2d07" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 33271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "347e4289f36f4aaa8fbfb0c17e79c8d1" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 46269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44abb8018a2b4cffa0fdfd882398b71f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 37821 } health_report { overall_health: UNKNOWN } } }
I20260928 23:53:41.064109 26460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:41.061000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41082 (local address 127.6.152.2:33271)
0928 23:53:41.061115 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:41.061118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:41.061132 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:53:41.061263 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:41.061266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:41.061314 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:41.061405 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:53:41.061651 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.062128 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.062898 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.063098 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.063168 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:41.063414 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:53:41.063779 (+ 365us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:53:41.063862 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:41.063912 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 23:53:41.063955 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 23:53:41.069643 26133 raft_consensus.cc:1261] T 42c94d3421fe48a9a371241c3eac9448 P 347e4289f36f4aaa8fbfb0c17e79c8d1 [term 1 FOLLOWER]: Refusing update from remote peer 60e4953c2af2441d9ea676247f9e2d07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:53:41.069679 26401 raft_consensus.cc:1261] T 42c94d3421fe48a9a371241c3eac9448 P 44abb8018a2b4cffa0fdfd882398b71f [term 1 FOLLOWER]: Refusing update from remote peer 60e4953c2af2441d9ea676247f9e2d07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:53:41.070327 26474 consensus_queue.cc:1035] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "347e4289f36f4aaa8fbfb0c17e79c8d1" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 46269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:53:41.070525 26470 consensus_queue.cc:1035] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44abb8018a2b4cffa0fdfd882398b71f" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 37821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:53:41.076278 26477 mvcc.cc:205] Tried to move back new op lower bound from 7334459887894507520 to 7334459887800946688. Current Snapshot: MvccSnapshot[applied={T|T < 7334459887894507520}]
I20260928 23:53:41.085199 25988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:53:41.085376 25988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:53:41.087524 25988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc3999ab948d42828abae24f101bbb4e: Sending DeleteTablet for 3 replicas of tablet 42c94d3421fe48a9a371241c3eac9448
I20260928 23:53:41.088265 26381 tablet_service.cc:1514] Processing DeleteTablet for tablet 42c94d3421fe48a9a371241c3eac9448 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:53:41 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:33310
I20260928 23:53:41.088263 26247 tablet_service.cc:1514] Processing DeleteTablet for tablet 42c94d3421fe48a9a371241c3eac9448 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:53:41 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:41076
I20260928 23:53:41.088728 26113 tablet_service.cc:1514] Processing DeleteTablet for tablet 42c94d3421fe48a9a371241c3eac9448 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:53:41 UTC) from {username='kudu', principal='kudu/127.6.152.62@KRBTEST.COM'} at 127.0.0.1:50216
I20260928 23:53:41.089421 26488 tablet_replica.cc:330] T 42c94d3421fe48a9a371241c3eac9448 P 347e4289f36f4aaa8fbfb0c17e79c8d1: stopping tablet replica
I20260928 23:53:41.089649 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 26050
I20260928 23:53:41.090372 26487 tablet_replica.cc:330] T 42c94d3421fe48a9a371241c3eac9448 P 44abb8018a2b4cffa0fdfd882398b71f: stopping tablet replica
I20260928 23:53:41.090643 26487 raft_consensus.cc:2229] T 42c94d3421fe48a9a371241c3eac9448 P 44abb8018a2b4cffa0fdfd882398b71f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:53:41.090932 26487 raft_consensus.cc:2258] T 42c94d3421fe48a9a371241c3eac9448 P 44abb8018a2b4cffa0fdfd882398b71f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:53:41.091307 26486 tablet_replica.cc:330] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07: stopping tablet replica
I20260928 23:53:41.091620 26486 raft_consensus.cc:2229] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:53:41.091991 26486 raft_consensus.cc:2258] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:53:41.092518 26487 ts_tablet_manager.cc:1905] T 42c94d3421fe48a9a371241c3eac9448 P 44abb8018a2b4cffa0fdfd882398b71f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:53:41.096005 26487 ts_tablet_manager.cc:1918] T 42c94d3421fe48a9a371241c3eac9448 P 44abb8018a2b4cffa0fdfd882398b71f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:53:41.096112 26487 log.cc:1196] T 42c94d3421fe48a9a371241c3eac9448 P 44abb8018a2b4cffa0fdfd882398b71f: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/42c94d3421fe48a9a371241c3eac9448
I20260928 23:53:41.096557 26487 ts_tablet_manager.cc:1939] T 42c94d3421fe48a9a371241c3eac9448 P 44abb8018a2b4cffa0fdfd882398b71f: Deleting consensus metadata
I20260928 23:53:41.097414 26486 ts_tablet_manager.cc:1905] T 42c94d3421fe48a9a371241c3eac9448 P 60e4953c2af2441d9ea676247f9e2d07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:53:41.097892 25975 catalog_manager.cc:4841] TS 44abb8018a2b4cffa0fdfd882398b71f (127.6.152.3:37821): tablet 42c94d3421fe48a9a371241c3eac9448 (table test-table [id=a4ed8c58593c42378e896c4aa1327f29]) successfully deleted
W20260928 23:53:41.098142 25974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.152.1:46269 (error 108)
I20260928 23:53:41.098266 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 26184
I20260928 23:53:41.100157 26458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:41.099850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:46269 (local address 127.0.0.1:50230)
0928 23:53:41.100011 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:41.100087 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.1:46269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 23:53:41.100411 25974 catalog_manager.cc:4584] TS 347e4289f36f4aaa8fbfb0c17e79c8d1 (127.6.152.1:46269): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 42c94d3421fe48a9a371241c3eac9448: Network error: Client connection negotiation failed: client connection to 127.6.152.1:46269: connect: Connection refused (error 111)
I20260928 23:53:41.104168 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 26318
I20260928 23:53:41.105042 26458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:41.104322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:33271 (local address 127.0.0.1:41094)
0928 23:53:41.104887 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:41.104951 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.2:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454}
I20260928 23:53:41.109447 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 25956
2026-09-28T23:53:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2198 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T23:53:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:53:41Z Disabled control of system clock
I20260928 23:53:41.139652 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:38541
--webserver_interface=127.6.152.62
--webserver_port=0
--builtin_ntp_servers=127.6.152.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:38541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:53:41.247285 26493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:41.247527 26493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:41.247560 26493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:41.251484 26493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:53:41.251538 26493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:41.251554 26493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:41.251570 26493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:53:41.251585 26493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:53:41.256362 26493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:38541
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:38541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:41.257448 26493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:41.258841 26493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:41.267073 26501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:41.267110 26499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:41.267100 26498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:41.267163 26493 server_base.cc:1034] running on GCE node
I20260928 23:53:41.267876 26493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:41.269016 26493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:41.270200 26493 hybrid_clock.cc:648] HybridClock initialized: now 1790639621270187 us; error 25 us; skew 500 ppm
I20260928 23:53:41.272035 26493 webserver.cc:460] Webserver started at http://127.6.152.62:41323/ using document root <none> and password file <none>
I20260928 23:53:41.272639 26493 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:41.272730 26493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:41.272989 26493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:41.275038 26493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "aee71f229b5b4da4b80c225966f062b4"
format_stamp: "Formatted at 2026-09-28 23:53:41 on dist-test-slave-p5bw"
server_key: "f0235fad371a4531b8cf2caea97beabb"
server_key_iv: "4bf200fd0d1ff6ba127f9600b37de583"
server_key_version: "encryptionkey@0"
I20260928 23:53:41.275581 26493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "aee71f229b5b4da4b80c225966f062b4"
format_stamp: "Formatted at 2026-09-28 23:53:41 on dist-test-slave-p5bw"
server_key: "f0235fad371a4531b8cf2caea97beabb"
server_key_iv: "4bf200fd0d1ff6ba127f9600b37de583"
server_key_version: "encryptionkey@0"
I20260928 23:53:41.279084 26493 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 23:53:41.281358 26507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:41.282534 26493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:53:41.282667 26493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "aee71f229b5b4da4b80c225966f062b4"
format_stamp: "Formatted at 2026-09-28 23:53:41 on dist-test-slave-p5bw"
server_key: "f0235fad371a4531b8cf2caea97beabb"
server_key_iv: "4bf200fd0d1ff6ba127f9600b37de583"
server_key_version: "encryptionkey@0"
I20260928 23:53:41.282773 26493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:41.300097 26493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:41.300755 26493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:41.300971 26493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:41.307369 26493 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:38541
I20260928 23:53:41.307379 26559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:38541 every 8 connection(s)
I20260928 23:53:41.308270 26493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:53:41.311267 26560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:41.315088 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 26493
I20260928 23:53:41.315229 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:53:41.315521 6752 external_mini_cluster.cc:1428] Setting key da0975871d306f1b92e506848351c091
I20260928 23:53:41.316780 26560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aee71f229b5b4da4b80c225966f062b4: Bootstrap starting.
I20260928 23:53:41.318996 26560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aee71f229b5b4da4b80c225966f062b4: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:41.319707 26560 log.cc:824] T 00000000000000000000000000000000 P aee71f229b5b4da4b80c225966f062b4: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:41.321488 26560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aee71f229b5b4da4b80c225966f062b4: No bootstrap required, opened a new log
I20260928 23:53:41.324566 26563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:41.316964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.6.152.62:38541)
0928 23:53:41.317578 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:41.317593 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:41.317627 (+ 34us) server_negotiation.cc:407] Connection header received
0928 23:53:41.318233 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:41.318248 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:41.318632 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:41.318957 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:41.319383 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.320380 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.321380 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.321889 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.322060 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:41.322918 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:41.322923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:41.322936 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:41.323316 (+ 380us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:41.323369 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:41.323388 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:41.323508 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:41.323622 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 23:53:41.323870 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":124,"threads_started":1}
I20260928 23:53:41.325009 26560 raft_consensus.cc:348] T 00000000000000000000000000000000 P aee71f229b5b4da4b80c225966f062b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee71f229b5b4da4b80c225966f062b4" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38541 } }
I20260928 23:53:41.325238 26560 raft_consensus.cc:374] T 00000000000000000000000000000000 P aee71f229b5b4da4b80c225966f062b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:41.325336 26560 raft_consensus.cc:729] T 00000000000000000000000000000000 P aee71f229b5b4da4b80c225966f062b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee71f229b5b4da4b80c225966f062b4, State: Initialized, Role: FOLLOWER
I20260928 23:53:41.325802 26560 consensus_queue.cc:260] T 00000000000000000000000000000000 P aee71f229b5b4da4b80c225966f062b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee71f229b5b4da4b80c225966f062b4" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38541 } }
I20260928 23:53:41.325933 26560 raft_consensus.cc:388] T 00000000000000000000000000000000 P aee71f229b5b4da4b80c225966f062b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:53:41.326015 26560 raft_consensus.cc:482] T 00000000000000000000000000000000 P aee71f229b5b4da4b80c225966f062b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:53:41.326121 26560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aee71f229b5b4da4b80c225966f062b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:41.327126 26560 raft_consensus.cc:504] T 00000000000000000000000000000000 P aee71f229b5b4da4b80c225966f062b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee71f229b5b4da4b80c225966f062b4" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38541 } }
I20260928 23:53:41.327461 26560 leader_election.cc:302] T 00000000000000000000000000000000 P aee71f229b5b4da4b80c225966f062b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aee71f229b5b4da4b80c225966f062b4; no voters:
I20260928 23:53:41.327771 26560 leader_election.cc:288] T 00000000000000000000000000000000 P aee71f229b5b4da4b80c225966f062b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:53:41.327875 26565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aee71f229b5b4da4b80c225966f062b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:53:41.328127 26565 raft_consensus.cc:686] T 00000000000000000000000000000000 P aee71f229b5b4da4b80c225966f062b4 [term 1 LEADER]: Becoming Leader. State: Replica: aee71f229b5b4da4b80c225966f062b4, State: Running, Role: LEADER
I20260928 23:53:41.328516 26565 consensus_queue.cc:237] T 00000000000000000000000000000000 P aee71f229b5b4da4b80c225966f062b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee71f229b5b4da4b80c225966f062b4" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38541 } }
I20260928 23:53:41.329105 26560 sys_catalog.cc:564] T 00000000000000000000000000000000 P aee71f229b5b4da4b80c225966f062b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:53:41.330107 26567 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee71f229b5b4da4b80c225966f062b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aee71f229b5b4da4b80c225966f062b4. Latest consensus state: current_term: 1 leader_uuid: "aee71f229b5b4da4b80c225966f062b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee71f229b5b4da4b80c225966f062b4" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38541 } } }
I20260928 23:53:41.330229 26567 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee71f229b5b4da4b80c225966f062b4 [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:41.330628 26574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:53:41.330719 26566 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee71f229b5b4da4b80c225966f062b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aee71f229b5b4da4b80c225966f062b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee71f229b5b4da4b80c225966f062b4" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 38541 } } }
I20260928 23:53:41.330873 26566 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee71f229b5b4da4b80c225966f062b4 [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:41.333758 26574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:53:41.338915 26574 catalog_manager.cc:1347] Generated new cluster ID: 5e412e0feecb4073a1c3b9a511437048
I20260928 23:53:41.338979 26574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:53:41.347394 26574 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:53:41.348551 26574 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:53:41.361160 26574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aee71f229b5b4da4b80c225966f062b4: Generated new TSK 0
I20260928 23:53:41.361869 26574 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:53:41.373778 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.1:0
--local_ip_for_outbound_sockets=127.6.152.1
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:38541
--builtin_ntp_servers=127.6.152.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:53:41.499935 26584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:41.500183 26584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:41.500222 26584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:41.504201 26584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:41.504256 26584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:41.504311 26584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.1
I20260928 23:53:41.508909 26584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.152.1
--webserver_port=0
--tserver_master_addrs=127.6.152.62:38541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.1
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:41.509956 26584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:41.511233 26584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:53:41.520794 26584 server_base.cc:1034] running on GCE node
W20260928 23:53:41.520735 26589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:41.520758 26592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:41.520741 26590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:41.521404 26584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:41.522014 26584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:41.523175 26584 hybrid_clock.cc:648] HybridClock initialized: now 1790639621523158 us; error 29 us; skew 500 ppm
I20260928 23:53:41.524781 26584 webserver.cc:460] Webserver started at http://127.6.152.1:34437/ using document root <none> and password file <none>
I20260928 23:53:41.525358 26584 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:41.525419 26584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:41.525578 26584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:41.527462 26584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-0/data/instance:
uuid: "e6828b8bdfc54264a9e3ce30cb0187dc"
format_stamp: "Formatted at 2026-09-28 23:53:41 on dist-test-slave-p5bw"
server_key: "490a7cb0da043febe633bd87683cedb8"
server_key_iv: "194cc9353bda4e90884258ac63f78b39"
server_key_version: "encryptionkey@0"
I20260928 23:53:41.527954 26584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance:
uuid: "e6828b8bdfc54264a9e3ce30cb0187dc"
format_stamp: "Formatted at 2026-09-28 23:53:41 on dist-test-slave-p5bw"
server_key: "490a7cb0da043febe633bd87683cedb8"
server_key_iv: "194cc9353bda4e90884258ac63f78b39"
server_key_version: "encryptionkey@0"
I20260928 23:53:41.531584 26584 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 23:53:41.533829 26598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:41.534970 26584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:53:41.535063 26584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-0/wal
uuid: "e6828b8bdfc54264a9e3ce30cb0187dc"
format_stamp: "Formatted at 2026-09-28 23:53:41 on dist-test-slave-p5bw"
server_key: "490a7cb0da043febe633bd87683cedb8"
server_key_iv: "194cc9353bda4e90884258ac63f78b39"
server_key_version: "encryptionkey@0"
I20260928 23:53:41.535143 26584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:41.557173 26584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:41.557786 26584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:41.557950 26584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:41.558455 26584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:53:41.559309 26584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:53:41.559353 26584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:41.559391 26584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:53:41.559406 26584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:41.568086 26584 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.1:36153
I20260928 23:53:41.568115 26711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.1:36153 every 8 connection(s)
I20260928 23:53:41.569037 26584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-0/data/info.pb
I20260928 23:53:41.570544 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 26584
I20260928 23:53:41.570688 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-0/wal/instance
I20260928 23:53:41.571038 6752 external_mini_cluster.cc:1428] Setting key 6320569af02e15c1cc1997ad4216c792
I20260928 23:53:41.573076 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.2:0
--local_ip_for_outbound_sockets=127.6.152.2
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:38541
--builtin_ntp_servers=127.6.152.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 23:53:41.578365 26563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:41.570777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:53657 (local address 127.6.152.62:38541)
0928 23:53:41.570930 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:41.570935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:41.572038 (+ 1103us) server_negotiation.cc:407] Connection header received
0928 23:53:41.573565 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:41.573572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:41.573635 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:41.573729 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:41.574723 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.575322 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.576450 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.576683 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.576793 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:41.577885 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:41.577897 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:41.577899 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:41.577911 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:41.577929 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:41.577936 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:41.578010 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:41.578168 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 23:53:41.578228 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 23:53:41.579159 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:41.571198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38541 (local address 127.6.152.1:53657)
0928 23:53:41.571881 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:41.571931 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:41.573327 (+ 1396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:41.573933 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:41.573956 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:41.574062 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:53:41.574559 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:41.574578 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.575456 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.575460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:41.576320 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:41.576329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.576850 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.576855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:41.577016 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:41.577642 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:53:41.577644 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:53:41.577682 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:53:41.577740 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:53:41.578024 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:53:41.578033 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:53:41.578039 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 23:53:41.578168 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 23:53:41.578443 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":135,"threads_started":1}
I20260928 23:53:41.580395 26712 heartbeater.cc:347] Connected to a master server at 127.6.152.62:38541
I20260928 23:53:41.580655 26712 heartbeater.cc:464] Registering TS with master...
I20260928 23:53:41.581324 26712 heartbeater.cc:511] Master 127.6.152.62:38541 requested a full tablet report, sending...
I20260928 23:53:41.582801 26524 ts_manager.cc:194] Registered new tserver with Master: e6828b8bdfc54264a9e3ce30cb0187dc (127.6.152.1:36153)
I20260928 23:53:41.583909 26524 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.152.1:53657
W20260928 23:53:41.682582 26715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:41.682819 26715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:41.682857 26715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:41.686920 26715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:41.686970 26715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:41.687053 26715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.2
I20260928 23:53:41.691788 26715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.152.2
--webserver_port=0
--tserver_master_addrs=127.6.152.62:38541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.2
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:41.692781 26715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:41.694104 26715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:41.703328 26720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:41.703331 26721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:41.703365 26723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:41.703399 26715 server_base.cc:1034] running on GCE node
I20260928 23:53:41.704042 26715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:41.704677 26715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:41.705842 26715 hybrid_clock.cc:648] HybridClock initialized: now 1790639621705805 us; error 46 us; skew 500 ppm
I20260928 23:53:41.707635 26715 webserver.cc:460] Webserver started at http://127.6.152.2:36465/ using document root <none> and password file <none>
I20260928 23:53:41.708199 26715 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:41.708262 26715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:41.708546 26715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:41.710594 26715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-1/data/instance:
uuid: "4234f00c85b74b48aae68ba845bd2e47"
format_stamp: "Formatted at 2026-09-28 23:53:41 on dist-test-slave-p5bw"
server_key: "eb62326103c0f1476ba1b98b92ade12b"
server_key_iv: "9a8d331498f253c29749602927a2e681"
server_key_version: "encryptionkey@0"
I20260928 23:53:41.711145 26715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance:
uuid: "4234f00c85b74b48aae68ba845bd2e47"
format_stamp: "Formatted at 2026-09-28 23:53:41 on dist-test-slave-p5bw"
server_key: "eb62326103c0f1476ba1b98b92ade12b"
server_key_iv: "9a8d331498f253c29749602927a2e681"
server_key_version: "encryptionkey@0"
I20260928 23:53:41.714699 26715 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 23:53:41.717015 26729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:41.718204 26715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:53:41.718299 26715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-1/wal
uuid: "4234f00c85b74b48aae68ba845bd2e47"
format_stamp: "Formatted at 2026-09-28 23:53:41 on dist-test-slave-p5bw"
server_key: "eb62326103c0f1476ba1b98b92ade12b"
server_key_iv: "9a8d331498f253c29749602927a2e681"
server_key_version: "encryptionkey@0"
I20260928 23:53:41.718379 26715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:41.739301 26715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:41.739907 26715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:41.740049 26715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:41.740567 26715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:53:41.741503 26715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:53:41.741552 26715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:41.741591 26715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:53:41.741611 26715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:41.750058 26715 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.2:41833
I20260928 23:53:41.750119 26842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.2:41833 every 8 connection(s)
I20260928 23:53:41.750953 26715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-1/data/info.pb
I20260928 23:53:41.759380 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 26715
I20260928 23:53:41.759522 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-1/wal/instance
I20260928 23:53:41.759842 6752 external_mini_cluster.cc:1428] Setting key c148184b29eadb6d418b93a1b887cb01
I20260928 23:53:41.759852 26563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:41.752668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.2:35073 (local address 127.6.152.62:38541)
0928 23:53:41.752840 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:41.752848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:41.753857 (+ 1009us) server_negotiation.cc:407] Connection header received
0928 23:53:41.755207 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:41.755210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:41.755269 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:41.755378 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:41.756228 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.756750 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.757927 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.758146 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.758256 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:41.759310 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:41.759313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:41.759315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:41.759327 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:41.759347 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:41.759355 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:41.759424 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:41.759663 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 23:53:41.759721 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 23:53:41.760766 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:41.753111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38541 (local address 127.6.152.2:35073)
0928 23:53:41.753714 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:41.753752 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:41.754994 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:41.755556 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:41.755562 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:41.755641 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:53:41.756088 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:41.756099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.756928 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.756934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:41.757798 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:41.757806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.758290 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.758294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:41.758459 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:41.759028 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:53:41.759030 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:53:41.759067 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:53:41.759144 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:53:41.759450 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:53:41.759460 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:53:41.759467 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 23:53:41.759640 (+ 173us) client_negotiation.cc:240] Negotiation successful
0928 23:53:41.759917 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":107,"threads_started":1}
I20260928 23:53:41.761858 26843 heartbeater.cc:347] Connected to a master server at 127.6.152.62:38541
I20260928 23:53:41.762044 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.152.3:0
--local_ip_for_outbound_sockets=127.6.152.3
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:38541
--builtin_ntp_servers=127.6.152.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 23:53:41.762080 26843 heartbeater.cc:464] Registering TS with master...
I20260928 23:53:41.762640 26843 heartbeater.cc:511] Master 127.6.152.62:38541 requested a full tablet report, sending...
I20260928 23:53:41.763705 26524 ts_manager.cc:194] Registered new tserver with Master: 4234f00c85b74b48aae68ba845bd2e47 (127.6.152.2:41833)
I20260928 23:53:41.764264 26524 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.152.2:35073
W20260928 23:53:41.876156 26846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:41.876539 26846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:41.876621 26846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:41.881731 26846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:41.881806 26846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:41.881903 26846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.152.3
I20260928 23:53:41.886910 26846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.152.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.152.3
--webserver_port=0
--tserver_master_addrs=127.6.152.62:38541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.152.3
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:41.887976 26846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:41.889356 26846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:41.898500 26851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:41.898485 26854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:41.898500 26852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:41.898766 26846 server_base.cc:1034] running on GCE node
I20260928 23:53:41.899266 26846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:41.899842 26846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:41.901046 26846 hybrid_clock.cc:648] HybridClock initialized: now 1790639621901035 us; error 31 us; skew 500 ppm
I20260928 23:53:41.902781 26846 webserver.cc:460] Webserver started at http://127.6.152.3:44221/ using document root <none> and password file <none>
I20260928 23:53:41.903369 26846 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:41.903456 26846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:41.903671 26846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:41.905691 26846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-2/data/instance:
uuid: "75108ef1a1ac418980a50dab61414014"
format_stamp: "Formatted at 2026-09-28 23:53:41 on dist-test-slave-p5bw"
server_key: "80d5adcba3799df0e41e6868508d6e49"
server_key_iv: "df8f08559f4e88fbca795119834a4998"
server_key_version: "encryptionkey@0"
I20260928 23:53:41.906237 26846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance:
uuid: "75108ef1a1ac418980a50dab61414014"
format_stamp: "Formatted at 2026-09-28 23:53:41 on dist-test-slave-p5bw"
server_key: "80d5adcba3799df0e41e6868508d6e49"
server_key_iv: "df8f08559f4e88fbca795119834a4998"
server_key_version: "encryptionkey@0"
I20260928 23:53:41.909665 26846 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260928 23:53:41.911966 26860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:41.913158 26846 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 23:53:41.913281 26846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-2/wal
uuid: "75108ef1a1ac418980a50dab61414014"
format_stamp: "Formatted at 2026-09-28 23:53:41 on dist-test-slave-p5bw"
server_key: "80d5adcba3799df0e41e6868508d6e49"
server_key_iv: "df8f08559f4e88fbca795119834a4998"
server_key_version: "encryptionkey@0"
I20260928 23:53:41.913384 26846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:41.932150 26846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:41.932802 26846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:41.933043 26846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:41.933602 26846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:53:41.934505 26846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:53:41.934580 26846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:41.934651 26846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:53:41.934701 26846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:41.943118 26846 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.3:34739
I20260928 23:53:41.943200 26973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.3:34739 every 8 connection(s)
I20260928 23:53:41.944080 26846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-2/data/info.pb
I20260928 23:53:41.948726 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 26846
I20260928 23:53:41.948872 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-2/wal/instance
I20260928 23:53:41.949159 6752 external_mini_cluster.cc:1428] Setting key aaff87e18953b7dace3442427aa74463
I20260928 23:53:41.953447 26563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:41.945861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.3:50921 (local address 127.6.152.62:38541)
0928 23:53:41.945999 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:41.946003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:41.947042 (+ 1039us) server_negotiation.cc:407] Connection header received
0928 23:53:41.948439 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:41.948442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:41.948504 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:41.948580 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:41.949788 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.950471 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.951639 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.951875 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.951985 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:41.952959 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:41.952963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:41.952965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:41.952977 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:41.953004 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:41.953014 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:41.953092 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:41.953247 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 23:53:41.953305 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:53:41.954211 26976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:41.946265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:38541 (local address 127.6.152.3:50921)
0928 23:53:41.946883 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:41.946931 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:41.948212 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:41.948744 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:41.948750 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:41.948896 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:53:41.949506 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:41.949526 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.950589 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.950592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:41.951521 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:41.951530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.952018 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.952022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:41.952177 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:41.952688 (+ 511us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:53:41.952689 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:53:41.952727 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:53:41.952785 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:53:41.953107 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:53:41.953116 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:53:41.953126 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 23:53:41.953243 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 23:53:41.953501 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":110,"threads_started":1}
I20260928 23:53:41.955176 26974 heartbeater.cc:347] Connected to a master server at 127.6.152.62:38541
I20260928 23:53:41.955372 26974 heartbeater.cc:464] Registering TS with master...
I20260928 23:53:41.955801 26974 heartbeater.cc:511] Master 127.6.152.62:38541 requested a full tablet report, sending...
I20260928 23:53:41.956681 26524 ts_manager.cc:194] Registered new tserver with Master: 75108ef1a1ac418980a50dab61414014 (127.6.152.3:34739)
I20260928 23:53:41.957242 26524 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.152.3:50921
I20260928 23:53:41.962801 6752 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:53:41.967953 26563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:41.964608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60230 (local address 127.6.152.62:38541)
0928 23:53:41.964731 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:41.964735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:41.964948 (+ 213us) server_negotiation.cc:407] Connection header received
0928 23:53:41.965041 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:41.965043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:41.965079 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:41.965143 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:41.965381 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.965864 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.966646 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.966842 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.966911 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:41.967661 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:41.967663 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:41.967666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:41.967674 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:41.967688 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:41.967695 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:41.967745 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:41.967818 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 23:53:41.967858 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:53:41.970772 26524 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60230:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:53:41.974236 26524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:53:41.987934 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:41.982814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:41833 (local address 127.0.0.1:43714)
0928 23:53:41.983495 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:41.983511 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:41.983604 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:41.984236 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:41.984239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:41.984252 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:41.984330 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:41.984339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.985333 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.985338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:41.986558 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:41.986567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.987400 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.987404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:41.987511 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:41.987547 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 23:53:41.987661 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 23:53:41.987774 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":17,"thread_start_us":86,"threads_started":1}
I20260928 23:53:41.987943 26990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:41.982965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43714 (local address 127.6.152.2:41833)
0928 23:53:41.983740 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:41.983753 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:41.983772 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:53:41.983837 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:41.983841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:41.983997 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:41.984116 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:41.984423 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.985224 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.986677 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.987278 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.987400 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:41.987583 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:41.987689 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 23:53:41.987774 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":94,"threads_started":1}
I20260928 23:53:41.989228 26985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:41.982107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:34739 (local address 127.0.0.1:53068)
0928 23:53:41.983225 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:41.983271 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:41.983421 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:41.983955 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:41.983962 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:41.983983 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:41.984118 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:41.984124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.985596 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.985599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:41.987475 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:41.987483 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.988681 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.988685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:41.988890 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:41.988931 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 23:53:41.989006 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 23:53:41.989100 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932,"mutex_wait_us":1176,"thread_start_us":74,"threads_started":1}
I20260928 23:53:41.989351 26989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:41.982883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39080 (local address 127.6.152.1:36153)
0928 23:53:41.983282 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:41.983300 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:41.983540 (+ 240us) server_negotiation.cc:407] Connection header received
0928 23:53:41.983629 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:41.983638 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:41.983841 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:41.983996 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:41.984384 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.985485 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.987621 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.988343 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.988450 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:41.988617 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:41.989101 (+ 484us) server_negotiation.cc:299] Negotiation successful
0928 23:53:41.989193 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":87,"threads_started":1}
I20260928 23:53:41.989655 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:41.982191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:36153 (local address 127.0.0.1:39080)
0928 23:53:41.983413 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:41.983426 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:41.983520 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:41.984149 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:41.984152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:41.984168 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:41.984263 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:41.984269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.985596 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.985599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:41.986576 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:41.986585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.988681 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.988685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:41.988932 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:41.988950 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 23:53:41.989009 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 23:53:41.989528 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1166,"mutex_wait_us":514,"thread_start_us":73,"threads_started":1}
I20260928 23:53:41.989691 26987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:41.982244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53068 (local address 127.6.152.3:34739)
0928 23:53:41.982604 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:41.982618 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:41.983572 (+ 954us) server_negotiation.cc:407] Connection header received
0928 23:53:41.983653 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:41.983657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:41.983807 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:41.983968 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:41.984226 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.985441 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.987587 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:41.988343 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:41.988453 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:41.988617 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:41.989449 (+ 832us) server_negotiation.cc:299] Negotiation successful
0928 23:53:41.989541 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":176,"threads_started":1}
I20260928 23:53:41.991351 26777 tablet_service.cc:1467] Processing CreateTablet for tablet 8bcc84a3190c45c7a24292e8754f30bf (DEFAULT_TABLE table=test-table [id=ef0a674e74dd4d26b088d442d5795629]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:53:41.992475 26777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bcc84a3190c45c7a24292e8754f30bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:41.992286 26646 tablet_service.cc:1467] Processing CreateTablet for tablet 8bcc84a3190c45c7a24292e8754f30bf (DEFAULT_TABLE table=test-table [id=ef0a674e74dd4d26b088d442d5795629]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:53:41.993304 26646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bcc84a3190c45c7a24292e8754f30bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:41.994881 26908 tablet_service.cc:1467] Processing CreateTablet for tablet 8bcc84a3190c45c7a24292e8754f30bf (DEFAULT_TABLE table=test-table [id=ef0a674e74dd4d26b088d442d5795629]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:53:41.995664 26908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bcc84a3190c45c7a24292e8754f30bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:41.999725 26991 tablet_bootstrap.cc:492] T 8bcc84a3190c45c7a24292e8754f30bf P 4234f00c85b74b48aae68ba845bd2e47: Bootstrap starting.
I20260928 23:53:42.000923 26992 tablet_bootstrap.cc:492] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc: Bootstrap starting.
I20260928 23:53:42.001442 26993 tablet_bootstrap.cc:492] T 8bcc84a3190c45c7a24292e8754f30bf P 75108ef1a1ac418980a50dab61414014: Bootstrap starting.
I20260928 23:53:42.002029 26991 tablet_bootstrap.cc:654] T 8bcc84a3190c45c7a24292e8754f30bf P 4234f00c85b74b48aae68ba845bd2e47: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:42.002839 26991 log.cc:824] T 8bcc84a3190c45c7a24292e8754f30bf P 4234f00c85b74b48aae68ba845bd2e47: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:42.003123 26992 tablet_bootstrap.cc:654] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:42.003508 26993 tablet_bootstrap.cc:654] T 8bcc84a3190c45c7a24292e8754f30bf P 75108ef1a1ac418980a50dab61414014: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:42.004029 26992 log.cc:824] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:42.004336 26993 log.cc:824] T 8bcc84a3190c45c7a24292e8754f30bf P 75108ef1a1ac418980a50dab61414014: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:42.004843 26991 tablet_bootstrap.cc:492] T 8bcc84a3190c45c7a24292e8754f30bf P 4234f00c85b74b48aae68ba845bd2e47: No bootstrap required, opened a new log
I20260928 23:53:42.005035 26991 ts_tablet_manager.cc:1397] T 8bcc84a3190c45c7a24292e8754f30bf P 4234f00c85b74b48aae68ba845bd2e47: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 23:53:42.005972 26993 tablet_bootstrap.cc:492] T 8bcc84a3190c45c7a24292e8754f30bf P 75108ef1a1ac418980a50dab61414014: No bootstrap required, opened a new log
I20260928 23:53:42.006008 26992 tablet_bootstrap.cc:492] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc: No bootstrap required, opened a new log
I20260928 23:53:42.006096 26993 ts_tablet_manager.cc:1397] T 8bcc84a3190c45c7a24292e8754f30bf P 75108ef1a1ac418980a50dab61414014: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 23:53:42.006168 26992 ts_tablet_manager.cc:1397] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 23:53:42.009034 26993 raft_consensus.cc:348] T 8bcc84a3190c45c7a24292e8754f30bf P 75108ef1a1ac418980a50dab61414014 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75108ef1a1ac418980a50dab61414014" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 34739 } } peers { permanent_uuid: "e6828b8bdfc54264a9e3ce30cb0187dc" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36153 } } peers { permanent_uuid: "4234f00c85b74b48aae68ba845bd2e47" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41833 } }
I20260928 23:53:42.009264 26993 raft_consensus.cc:374] T 8bcc84a3190c45c7a24292e8754f30bf P 75108ef1a1ac418980a50dab61414014 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:42.009333 26993 raft_consensus.cc:729] T 8bcc84a3190c45c7a24292e8754f30bf P 75108ef1a1ac418980a50dab61414014 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75108ef1a1ac418980a50dab61414014, State: Initialized, Role: FOLLOWER
I20260928 23:53:42.009254 26991 raft_consensus.cc:348] T 8bcc84a3190c45c7a24292e8754f30bf P 4234f00c85b74b48aae68ba845bd2e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75108ef1a1ac418980a50dab61414014" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 34739 } } peers { permanent_uuid: "e6828b8bdfc54264a9e3ce30cb0187dc" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36153 } } peers { permanent_uuid: "4234f00c85b74b48aae68ba845bd2e47" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41833 } }
I20260928 23:53:42.009521 26991 raft_consensus.cc:374] T 8bcc84a3190c45c7a24292e8754f30bf P 4234f00c85b74b48aae68ba845bd2e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:42.009596 26991 raft_consensus.cc:729] T 8bcc84a3190c45c7a24292e8754f30bf P 4234f00c85b74b48aae68ba845bd2e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4234f00c85b74b48aae68ba845bd2e47, State: Initialized, Role: FOLLOWER
I20260928 23:53:42.009888 26993 consensus_queue.cc:260] T 8bcc84a3190c45c7a24292e8754f30bf P 75108ef1a1ac418980a50dab61414014 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75108ef1a1ac418980a50dab61414014" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 34739 } } peers { permanent_uuid: "e6828b8bdfc54264a9e3ce30cb0187dc" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36153 } } peers { permanent_uuid: "4234f00c85b74b48aae68ba845bd2e47" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41833 } }
I20260928 23:53:42.010080 26991 consensus_queue.cc:260] T 8bcc84a3190c45c7a24292e8754f30bf P 4234f00c85b74b48aae68ba845bd2e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75108ef1a1ac418980a50dab61414014" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 34739 } } peers { permanent_uuid: "e6828b8bdfc54264a9e3ce30cb0187dc" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36153 } } peers { permanent_uuid: "4234f00c85b74b48aae68ba845bd2e47" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41833 } }
I20260928 23:53:42.010221 26992 raft_consensus.cc:348] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75108ef1a1ac418980a50dab61414014" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 34739 } } peers { permanent_uuid: "e6828b8bdfc54264a9e3ce30cb0187dc" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36153 } } peers { permanent_uuid: "4234f00c85b74b48aae68ba845bd2e47" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41833 } }
I20260928 23:53:42.010522 26992 raft_consensus.cc:374] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:42.010618 26992 raft_consensus.cc:729] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6828b8bdfc54264a9e3ce30cb0187dc, State: Initialized, Role: FOLLOWER
I20260928 23:53:42.010675 26974 heartbeater.cc:503] Master 127.6.152.62:38541 was elected leader, sending a full tablet report...
I20260928 23:53:42.011039 26993 ts_tablet_manager.cc:1428] T 8bcc84a3190c45c7a24292e8754f30bf P 75108ef1a1ac418980a50dab61414014: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 23:53:42.011049 26991 ts_tablet_manager.cc:1428] T 8bcc84a3190c45c7a24292e8754f30bf P 4234f00c85b74b48aae68ba845bd2e47: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:53:42.011070 26992 consensus_queue.cc:260] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75108ef1a1ac418980a50dab61414014" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 34739 } } peers { permanent_uuid: "e6828b8bdfc54264a9e3ce30cb0187dc" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36153 } } peers { permanent_uuid: "4234f00c85b74b48aae68ba845bd2e47" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41833 } }
I20260928 23:53:42.011356 26843 heartbeater.cc:503] Master 127.6.152.62:38541 was elected leader, sending a full tablet report...
I20260928 23:53:42.012210 26712 heartbeater.cc:503] Master 127.6.152.62:38541 was elected leader, sending a full tablet report...
I20260928 23:53:42.012454 26992 ts_tablet_manager.cc:1428] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
W20260928 23:53:42.070202 26713 tablet.cc:2369] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:53:42.080432 26999 raft_consensus.cc:482] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:53:42.080664 26999 raft_consensus.cc:504] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75108ef1a1ac418980a50dab61414014" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 34739 } } peers { permanent_uuid: "e6828b8bdfc54264a9e3ce30cb0187dc" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36153 } } peers { permanent_uuid: "4234f00c85b74b48aae68ba845bd2e47" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41833 } }
I20260928 23:53:42.081802 26999 leader_election.cc:288] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75108ef1a1ac418980a50dab61414014 (127.6.152.3:34739), 4234f00c85b74b48aae68ba845bd2e47 (127.6.152.2:41833)
I20260928 23:53:42.086910 26990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:42.082557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:45959 (local address 127.6.152.2:41833)
0928 23:53:42.082685 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:42.082689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:42.082817 (+ 128us) server_negotiation.cc:407] Connection header received
0928 23:53:42.082896 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:42.082898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:42.082944 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:42.083000 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:42.083274 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:42.083988 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:42.085591 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:42.086210 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:42.086318 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:42.086377 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:42.086690 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 23:53:42.086767 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 23:53:42.086897 26987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:42.082163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.152.1:52195 (local address 127.6.152.3:34739)
0928 23:53:42.082311 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:42.082317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:42.082412 (+ 95us) server_negotiation.cc:407] Connection header received
0928 23:53:42.082611 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:42.082615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:42.082676 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:42.082741 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:53:42.083094 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:42.083876 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:42.085608 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:42.086210 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:42.086318 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:42.086377 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:42.086707 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 23:53:42.086776 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 23:53:42.086948 27000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:42.082027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.3:34739 (local address 127.6.152.1:52195)
0928 23:53:42.082311 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:42.082326 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:42.082475 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:42.082767 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:42.082773 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:42.082799 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:42.082958 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:42.082963 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:42.083988 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:42.083990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:42.085444 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:42.085455 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:42.086334 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:42.086338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:42.086536 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:42.086578 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 23:53:42.086657 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 23:53:42.086746 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":142,"thread_start_us":93,"threads_started":1}
I20260928 23:53:42.086982 27001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:42.082427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.2:41833 (local address 127.6.152.1:45959)
0928 23:53:42.082725 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:42.082740 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:53:42.082815 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:53:42.083033 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:53:42.083038 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:53:42.083061 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:53:42.083159 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:42.083163 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:42.084076 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:42.084079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:42.085444 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:53:42.085455 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:53:42.086337 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:53:42.086340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:53:42.086531 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:42.086582 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 23:53:42.086657 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 23:53:42.086746 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":118,"thread_start_us":94,"threads_started":1}
I20260928 23:53:42.087559 26797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bcc84a3190c45c7a24292e8754f30bf" candidate_uuid: "e6828b8bdfc54264a9e3ce30cb0187dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4234f00c85b74b48aae68ba845bd2e47" is_pre_election: true
I20260928 23:53:42.087764 26928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bcc84a3190c45c7a24292e8754f30bf" candidate_uuid: "e6828b8bdfc54264a9e3ce30cb0187dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75108ef1a1ac418980a50dab61414014" is_pre_election: true
I20260928 23:53:42.087859 26797 raft_consensus.cc:2454] T 8bcc84a3190c45c7a24292e8754f30bf P 4234f00c85b74b48aae68ba845bd2e47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6828b8bdfc54264a9e3ce30cb0187dc in term 0.
I20260928 23:53:42.088054 26928 raft_consensus.cc:2454] T 8bcc84a3190c45c7a24292e8754f30bf P 75108ef1a1ac418980a50dab61414014 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6828b8bdfc54264a9e3ce30cb0187dc in term 0.
I20260928 23:53:42.088413 26602 leader_election.cc:302] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4234f00c85b74b48aae68ba845bd2e47, e6828b8bdfc54264a9e3ce30cb0187dc; no voters:
I20260928 23:53:42.088634 26999 raft_consensus.cc:2781] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:53:42.088712 26999 raft_consensus.cc:482] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:53:42.088742 26999 raft_consensus.cc:3037] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:42.089782 26999 raft_consensus.cc:504] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75108ef1a1ac418980a50dab61414014" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 34739 } } peers { permanent_uuid: "e6828b8bdfc54264a9e3ce30cb0187dc" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36153 } } peers { permanent_uuid: "4234f00c85b74b48aae68ba845bd2e47" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41833 } }
I20260928 23:53:42.090121 26999 leader_election.cc:288] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc [CANDIDATE]: Term 1 election: Requested vote from peers 75108ef1a1ac418980a50dab61414014 (127.6.152.3:34739), 4234f00c85b74b48aae68ba845bd2e47 (127.6.152.2:41833)
I20260928 23:53:42.090464 26928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bcc84a3190c45c7a24292e8754f30bf" candidate_uuid: "e6828b8bdfc54264a9e3ce30cb0187dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75108ef1a1ac418980a50dab61414014"
I20260928 23:53:42.090512 26797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bcc84a3190c45c7a24292e8754f30bf" candidate_uuid: "e6828b8bdfc54264a9e3ce30cb0187dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4234f00c85b74b48aae68ba845bd2e47"
I20260928 23:53:42.090561 26928 raft_consensus.cc:3037] T 8bcc84a3190c45c7a24292e8754f30bf P 75108ef1a1ac418980a50dab61414014 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:42.090658 26797 raft_consensus.cc:3037] T 8bcc84a3190c45c7a24292e8754f30bf P 4234f00c85b74b48aae68ba845bd2e47 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:42.091552 26928 raft_consensus.cc:2454] T 8bcc84a3190c45c7a24292e8754f30bf P 75108ef1a1ac418980a50dab61414014 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6828b8bdfc54264a9e3ce30cb0187dc in term 1.
I20260928 23:53:42.091719 26797 raft_consensus.cc:2454] T 8bcc84a3190c45c7a24292e8754f30bf P 4234f00c85b74b48aae68ba845bd2e47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6828b8bdfc54264a9e3ce30cb0187dc in term 1.
I20260928 23:53:42.091889 26602 leader_election.cc:302] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75108ef1a1ac418980a50dab61414014, e6828b8bdfc54264a9e3ce30cb0187dc; no voters:
I20260928 23:53:42.092063 26999 raft_consensus.cc:2781] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:53:42.092275 26999 raft_consensus.cc:686] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc [term 1 LEADER]: Becoming Leader. State: Replica: e6828b8bdfc54264a9e3ce30cb0187dc, State: Running, Role: LEADER
I20260928 23:53:42.092620 26999 consensus_queue.cc:237] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75108ef1a1ac418980a50dab61414014" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 34739 } } peers { permanent_uuid: "e6828b8bdfc54264a9e3ce30cb0187dc" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36153 } } peers { permanent_uuid: "4234f00c85b74b48aae68ba845bd2e47" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41833 } }
I20260928 23:53:42.095727 26523 catalog_manager.cc:5495] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc reported cstate change: term changed from 0 to 1, leader changed from <none> to e6828b8bdfc54264a9e3ce30cb0187dc (127.6.152.1). New cstate: current_term: 1 leader_uuid: "e6828b8bdfc54264a9e3ce30cb0187dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75108ef1a1ac418980a50dab61414014" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 34739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6828b8bdfc54264a9e3ce30cb0187dc" member_type: VOTER last_known_addr { host: "127.6.152.1" port: 36153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4234f00c85b74b48aae68ba845bd2e47" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41833 } health_report { overall_health: UNKNOWN } } }
I20260928 23:53:42.107664 26989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:42.104161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.6.152.1:36153)
0928 23:53:42.104276 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:42.104281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:42.104323 (+ 42us) server_negotiation.cc:407] Connection header received
0928 23:53:42.104514 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:42.104516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:42.104568 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:42.104670 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:53:42.104963 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:42.105444 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:42.106281 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:42.106503 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:42.106612 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:42.106905 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:53:42.107286 (+ 381us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:53:42.107414 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:42.107479 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 23:53:42.107528 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:53:42.113067 26928 raft_consensus.cc:1261] T 8bcc84a3190c45c7a24292e8754f30bf P 75108ef1a1ac418980a50dab61414014 [term 1 FOLLOWER]: Refusing update from remote peer e6828b8bdfc54264a9e3ce30cb0187dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:53:42.113077 26797 raft_consensus.cc:1261] T 8bcc84a3190c45c7a24292e8754f30bf P 4234f00c85b74b48aae68ba845bd2e47 [term 1 FOLLOWER]: Refusing update from remote peer e6828b8bdfc54264a9e3ce30cb0187dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:53:42.113718 26999 consensus_queue.cc:1035] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "75108ef1a1ac418980a50dab61414014" member_type: VOTER last_known_addr { host: "127.6.152.3" port: 34739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:53:42.113897 27002 consensus_queue.cc:1035] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4234f00c85b74b48aae68ba845bd2e47" member_type: VOTER last_known_addr { host: "127.6.152.2" port: 41833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:53:42.122861 27007 mvcc.cc:205] Tried to move back new op lower bound from 7334459892168269824 to 7334459892093939712. Current Snapshot: MvccSnapshot[applied={T|T < 7334459892168269824}]
I20260928 23:53:42.124171 27006 mvcc.cc:205] Tried to move back new op lower bound from 7334459892168269824 to 7334459892093939712. Current Snapshot: MvccSnapshot[applied={T|T < 7334459892168269824}]
I20260928 23:53:42.129488 26522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:53:42.129663 26522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:53:42.132089 26522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aee71f229b5b4da4b80c225966f062b4: Sending DeleteTablet for 3 replicas of tablet 8bcc84a3190c45c7a24292e8754f30bf
I20260928 23:53:42.132843 26908 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bcc84a3190c45c7a24292e8754f30bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:53:42 UTC) from {username='slave'} at 127.0.0.1:53068
I20260928 23:53:42.132798 26646 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bcc84a3190c45c7a24292e8754f30bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:53:42 UTC) from {username='slave'} at 127.0.0.1:39080
I20260928 23:53:42.132906 26777 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bcc84a3190c45c7a24292e8754f30bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:53:42 UTC) from {username='slave'} at 127.0.0.1:43714
I20260928 23:53:42.133364 27016 tablet_replica.cc:330] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc: stopping tablet replica
I20260928 23:53:42.133358 27015 tablet_replica.cc:330] T 8bcc84a3190c45c7a24292e8754f30bf P 75108ef1a1ac418980a50dab61414014: stopping tablet replica
I20260928 23:53:42.133670 27015 raft_consensus.cc:2229] T 8bcc84a3190c45c7a24292e8754f30bf P 75108ef1a1ac418980a50dab61414014 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:53:42.133731 27016 raft_consensus.cc:2229] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:53:42.134059 27015 raft_consensus.cc:2258] T 8bcc84a3190c45c7a24292e8754f30bf P 75108ef1a1ac418980a50dab61414014 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:53:42.134112 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 26584
I20260928 23:53:42.134117 27016 raft_consensus.cc:2258] T 8bcc84a3190c45c7a24292e8754f30bf P e6828b8bdfc54264a9e3ce30cb0187dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:53:42.136236 27017 tablet_replica.cc:330] T 8bcc84a3190c45c7a24292e8754f30bf P 4234f00c85b74b48aae68ba845bd2e47: stopping tablet replica
I20260928 23:53:42.136369 27015 ts_tablet_manager.cc:1905] T 8bcc84a3190c45c7a24292e8754f30bf P 75108ef1a1ac418980a50dab61414014: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:53:42.136554 27017 raft_consensus.cc:2229] T 8bcc84a3190c45c7a24292e8754f30bf P 4234f00c85b74b48aae68ba845bd2e47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:53:42.136865 27017 raft_consensus.cc:2258] T 8bcc84a3190c45c7a24292e8754f30bf P 4234f00c85b74b48aae68ba845bd2e47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:53:42.137989 27017 ts_tablet_manager.cc:1905] T 8bcc84a3190c45c7a24292e8754f30bf P 4234f00c85b74b48aae68ba845bd2e47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:53:42.139834 27015 ts_tablet_manager.cc:1918] T 8bcc84a3190c45c7a24292e8754f30bf P 75108ef1a1ac418980a50dab61414014: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:53:42.139948 27015 log.cc:1196] T 8bcc84a3190c45c7a24292e8754f30bf P 75108ef1a1ac418980a50dab61414014: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-2/wal/wals/8bcc84a3190c45c7a24292e8754f30bf
I20260928 23:53:42.140103 27017 ts_tablet_manager.cc:1918] T 8bcc84a3190c45c7a24292e8754f30bf P 4234f00c85b74b48aae68ba845bd2e47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:53:42.140206 27017 log.cc:1196] T 8bcc84a3190c45c7a24292e8754f30bf P 4234f00c85b74b48aae68ba845bd2e47: Deleting WAL directory at /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639374546653-6752-0/minicluster-data/ts-1/wal/wals/8bcc84a3190c45c7a24292e8754f30bf
I20260928 23:53:42.140278 27015 ts_tablet_manager.cc:1939] T 8bcc84a3190c45c7a24292e8754f30bf P 75108ef1a1ac418980a50dab61414014: Deleting consensus metadata
I20260928 23:53:42.140525 27017 ts_tablet_manager.cc:1939] T 8bcc84a3190c45c7a24292e8754f30bf P 4234f00c85b74b48aae68ba845bd2e47: Deleting consensus metadata
I20260928 23:53:42.141571 26511 catalog_manager.cc:4841] TS 4234f00c85b74b48aae68ba845bd2e47 (127.6.152.2:41833): tablet 8bcc84a3190c45c7a24292e8754f30bf (table test-table [id=ef0a674e74dd4d26b088d442d5795629]) successfully deleted
I20260928 23:53:42.141775 26511 catalog_manager.cc:4841] TS 75108ef1a1ac418980a50dab61414014 (127.6.152.3:34739): tablet 8bcc84a3190c45c7a24292e8754f30bf (table test-table [id=ef0a674e74dd4d26b088d442d5795629]) successfully deleted
W20260928 23:53:42.141907 26509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.152.1:36153 (error 108)
I20260928 23:53:42.142637 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 26715
I20260928 23:53:42.142879 26985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:42.142555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.1:36153 (local address 127.0.0.1:39092)
0928 23:53:42.142706 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 23:53:42.142783 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.1:36153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 23:53:42.143704 26509 catalog_manager.cc:4584] TS e6828b8bdfc54264a9e3ce30cb0187dc (127.6.152.1:36153): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8bcc84a3190c45c7a24292e8754f30bf: Network error: Client connection negotiation failed: client connection to 127.6.152.1:36153: connect: Connection refused (error 111)
I20260928 23:53:42.148221 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 26846
I20260928 23:53:42.153486 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 26493
2026-09-28T23:53:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1042 ms)
[----------] 6 tests from EncryptionPolicyTest (9969 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:53:42 dist-test-slave-p5bw krb5kdc[27021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:53:42 dist-test-slave-p5bw krb5kdc[27021](info): set up 2 sockets
Sep 28 23:53:42 dist-test-slave-p5bw krb5kdc[27021](info): commencing operation
krb5kdc: starting...
W20260928 23:53:43.193640 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:53:43 dist-test-slave-p5bw krb5kdc[27021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:53:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:53:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:53:43.316243 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37883
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:53:43.424919 27034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:43.425177 27034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:43.425258 27034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:43.429236 27034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:53:43.429294 27034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:43.429313 27034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:43.429330 27034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:53:43.429378 27034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:53:43.434705 27034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37883
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:43.435755 27034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:43.436784 27034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:43.445058 27040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:43.445086 27039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:43.445152 27034 server_base.cc:1034] running on GCE node
W20260928 23:53:43.445086 27042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:43.445716 27034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:43.446769 27034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:43.447932 27034 hybrid_clock.cc:648] HybridClock initialized: now 1790639623447920 us; error 26 us; skew 500 ppm
Sep 28 23:53:43 dist-test-slave-p5bw krb5kdc[27021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:43.450681 27034 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:53:43.451452 27034 webserver.cc:460] Webserver started at http://127.0.0.1:32821/ using document root <none> and password file <none>
I20260928 23:53:43.452003 27034 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:43.452090 27034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:43.452314 27034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:43.454315 27034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "94af5ece945941a69d63164660c56090"
format_stamp: "Formatted at 2026-09-28 23:53:43 on dist-test-slave-p5bw"
server_key: "845d9d5ea6f8b81a7b7f8fb5c19b10dc"
server_key_iv: "f47c946d5495d9b111114c323c649ae9"
server_key_version: "encryptionkey@0"
I20260928 23:53:43.454852 27034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "94af5ece945941a69d63164660c56090"
format_stamp: "Formatted at 2026-09-28 23:53:43 on dist-test-slave-p5bw"
server_key: "845d9d5ea6f8b81a7b7f8fb5c19b10dc"
server_key_iv: "f47c946d5495d9b111114c323c649ae9"
server_key_version: "encryptionkey@0"
I20260928 23:53:43.458403 27034 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 23:53:43.461197 27049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:43.462345 27034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:53:43.462484 27034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "94af5ece945941a69d63164660c56090"
format_stamp: "Formatted at 2026-09-28 23:53:43 on dist-test-slave-p5bw"
server_key: "845d9d5ea6f8b81a7b7f8fb5c19b10dc"
server_key_iv: "f47c946d5495d9b111114c323c649ae9"
server_key_version: "encryptionkey@0"
I20260928 23:53:43.462595 27034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:43.489934 27034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:43.493510 27034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:43.493726 27034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:43.500511 27034 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37883
I20260928 23:53:43.500592 27101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37883 every 8 connection(s)
I20260928 23:53:43.501528 27034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:53:43.502568 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 27034
I20260928 23:53:43.502707 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:53:43.503039 6752 external_mini_cluster.cc:1428] Setting key ae77b7748cd292305155a59febb13af6
I20260928 23:53:43.504840 27102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:43.510205 27102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94af5ece945941a69d63164660c56090: Bootstrap starting.
Sep 28 23:53:43 dist-test-slave-p5bw krb5kdc[27021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 23:53:43.512404 27102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94af5ece945941a69d63164660c56090: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:43.513185 27102 log.cc:824] T 00000000000000000000000000000000 P 94af5ece945941a69d63164660c56090: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:43.515025 27102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94af5ece945941a69d63164660c56090: No bootstrap required, opened a new log
I20260928 23:53:43.516934 27105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:43.504479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51167 (local address 127.0.0.1:37883)
0928 23:53:43.504992 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:43.505010 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:43.505045 (+ 35us) server_negotiation.cc:407] Connection header received
0928 23:53:43.505672 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:43.505687 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:43.506012 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:43.506340 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:43.507147 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:43.508141 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:43.509065 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:43.509525 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:43.509699 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:43.512176 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:43.512181 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:43.512193 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:53:43.512219 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:43.514157 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:43.514669 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:43.514673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:43.514678 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:43.514776 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:43.515111 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:43.515114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:43.515115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:43.515490 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:43.515706 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:43.515992 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 23:53:43.516204 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":131,"threads_started":1}
I20260928 23:53:43.518671 27102 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94af5ece945941a69d63164660c56090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94af5ece945941a69d63164660c56090" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37883 } }
I20260928 23:53:43.518942 27102 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94af5ece945941a69d63164660c56090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:43.519068 27102 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94af5ece945941a69d63164660c56090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94af5ece945941a69d63164660c56090, State: Initialized, Role: FOLLOWER
I20260928 23:53:43.519546 27102 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94af5ece945941a69d63164660c56090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94af5ece945941a69d63164660c56090" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37883 } }
I20260928 23:53:43.519696 27102 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94af5ece945941a69d63164660c56090 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:53:43.519764 27102 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94af5ece945941a69d63164660c56090 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:53:43.519804 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:53:43.519865 27102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94af5ece945941a69d63164660c56090 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:43.521025 27102 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94af5ece945941a69d63164660c56090 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94af5ece945941a69d63164660c56090" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37883 } }
I20260928 23:53:43.521407 27102 leader_election.cc:302] T 00000000000000000000000000000000 P 94af5ece945941a69d63164660c56090 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94af5ece945941a69d63164660c56090; no voters:
I20260928 23:53:43.521732 27102 leader_election.cc:288] T 00000000000000000000000000000000 P 94af5ece945941a69d63164660c56090 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:53:43.521831 27111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94af5ece945941a69d63164660c56090 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:53:43.522058 27111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94af5ece945941a69d63164660c56090 [term 1 LEADER]: Becoming Leader. State: Replica: 94af5ece945941a69d63164660c56090, State: Running, Role: LEADER
I20260928 23:53:43.522444 27111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94af5ece945941a69d63164660c56090 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94af5ece945941a69d63164660c56090" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37883 } }
I20260928 23:53:43.522992 27102 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94af5ece945941a69d63164660c56090 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:53:43.524134 27112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94af5ece945941a69d63164660c56090 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94af5ece945941a69d63164660c56090. Latest consensus state: current_term: 1 leader_uuid: "94af5ece945941a69d63164660c56090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94af5ece945941a69d63164660c56090" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37883 } } }
I20260928 23:53:43.524243 27112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94af5ece945941a69d63164660c56090 [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:43.524616 27113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94af5ece945941a69d63164660c56090 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94af5ece945941a69d63164660c56090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94af5ece945941a69d63164660c56090" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37883 } } }
I20260928 23:53:43.524770 27113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94af5ece945941a69d63164660c56090 [sys.catalog]: This master's current role is: LEADER
W20260928 23:53:43.528165 27127 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94af5ece945941a69d63164660c56090: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:53:43.528244 27127 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:53:43.528350 27121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:53:43.529152 27121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:53:43.530725 27105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:43.522078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46175 (local address 127.0.0.1:37883)
0928 23:53:43.522182 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:43.522186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:43.522495 (+ 309us) server_negotiation.cc:407] Connection header received
0928 23:53:43.522670 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:43.522673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:43.522714 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:43.522759 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:43.523677 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:43.524490 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:43.525792 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:43.526038 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:43.526146 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:43.527467 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:43.527471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:43.527474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:53:43.527497 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:43.528958 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:43.529473 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:43.529477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:43.529479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:43.529557 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:43.529876 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:43.529879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:43.529880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:43.530050 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:43.530197 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:43.530467 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 23:53:43.530555 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 23:53:43.534727 27121 catalog_manager.cc:1347] Generated new cluster ID: 2a2326c0ba944c4880b9bd3f875902fc
I20260928 23:53:43.534790 27121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:53:43.555471 27121 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:53:43.556743 27121 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:53:43.564373 27121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94af5ece945941a69d63164660c56090: Generated new TSK 0
I20260928 23:53:43.565219 27121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:53:43.578670 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 27034
W20260928 23:53:43.584275 27019 connection.cc:535] client connection to 127.0.0.1:37883 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37883): Connection reset by peer (error 104)
2026-09-28T23:53:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:53:43 dist-test-slave-p5bw krb5kdc[27133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:53:43 dist-test-slave-p5bw krb5kdc[27133](info): set up 2 sockets
Sep 28 23:53:43 dist-test-slave-p5bw krb5kdc[27133](info): commencing operation
krb5kdc: starting...
W20260928 23:53:44.613663 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:53:44 dist-test-slave-p5bw krb5kdc[27133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:53:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:53:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:53:44.733654 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40569
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:53:44.839789 27146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:44.840212 27146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:44.840370 27146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:44.844867 27146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:53:44.844959 27146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:44.845002 27146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:44.845043 27146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:53:44.845077 27146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:53:44.850507 27146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40569
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:44.851575 27146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:44.852596 27146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:44.861070 27152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:44.861063 27151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:44.861063 27154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:44.861387 27146 server_base.cc:1034] running on GCE node
I20260928 23:53:44.862025 27146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:44.863082 27146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:44.864243 27146 hybrid_clock.cc:648] HybridClock initialized: now 1790639624864209 us; error 46 us; skew 500 ppm
Sep 28 23:53:44 dist-test-slave-p5bw krb5kdc[27133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:44.866925 27146 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:53:44.867767 27146 webserver.cc:460] Webserver started at http://127.0.0.1:34447/ using document root <none> and password file <none>
I20260928 23:53:44.868315 27146 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:44.868402 27146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:44.868626 27146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:44.870687 27146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "8ee490844e6c41ad97576ef04bbee24b"
format_stamp: "Formatted at 2026-09-28 23:53:44 on dist-test-slave-p5bw"
server_key: "520a1f6be0b19aeedea11a9fd3fd3209"
server_key_iv: "ca9a7474edfcd4a68aabef6338f616af"
server_key_version: "encryptionkey@0"
I20260928 23:53:44.871272 27146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "8ee490844e6c41ad97576ef04bbee24b"
format_stamp: "Formatted at 2026-09-28 23:53:44 on dist-test-slave-p5bw"
server_key: "520a1f6be0b19aeedea11a9fd3fd3209"
server_key_iv: "ca9a7474edfcd4a68aabef6338f616af"
server_key_version: "encryptionkey@0"
I20260928 23:53:44.874852 27146 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260928 23:53:44.877128 27161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:44.878391 27146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:53:44.878540 27146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "8ee490844e6c41ad97576ef04bbee24b"
format_stamp: "Formatted at 2026-09-28 23:53:44 on dist-test-slave-p5bw"
server_key: "520a1f6be0b19aeedea11a9fd3fd3209"
server_key_iv: "ca9a7474edfcd4a68aabef6338f616af"
server_key_version: "encryptionkey@0"
I20260928 23:53:44.878654 27146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:44.891845 27146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:44.894649 27146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:44.894851 27146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:44.902206 27146 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40569
I20260928 23:53:44.902235 27213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40569 every 8 connection(s)
I20260928 23:53:44.903213 27146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:53:44.905936 27214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:44.909260 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 27146
I20260928 23:53:44.909415 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:53:44.909725 6752 external_mini_cluster.cc:1428] Setting key 78203541ca9bb0c4f48b30b5f9d71823
I20260928 23:53:44.911959 27214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee490844e6c41ad97576ef04bbee24b: Bootstrap starting.
I20260928 23:53:44.914732 27214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ee490844e6c41ad97576ef04bbee24b: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:44.915582 27214 log.cc:824] T 00000000000000000000000000000000 P 8ee490844e6c41ad97576ef04bbee24b: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:44.917858 27214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee490844e6c41ad97576ef04bbee24b: No bootstrap required, opened a new log
I20260928 23:53:44.920653 27217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:44.911182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49265 (local address 127.0.0.1:40569)
0928 23:53:44.911629 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:44.911648 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:44.911690 (+ 42us) server_negotiation.cc:407] Connection header received
0928 23:53:44.912528 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:44.912552 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:44.913005 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:44.913417 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:44.913838 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:44.915320 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:44.916503 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:44.917196 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:44.917362 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:44.918380 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:44.918387 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:44.918412 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:44.918893 (+ 481us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:44.918963 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:44.918991 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:44.919116 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:44.919329 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 23:53:44.919472 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":120,"threads_started":1}
I20260928 23:53:44.921646 27214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ee490844e6c41ad97576ef04bbee24b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee490844e6c41ad97576ef04bbee24b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40569 } }
I20260928 23:53:44.921916 27214 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ee490844e6c41ad97576ef04bbee24b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:44.921983 27214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ee490844e6c41ad97576ef04bbee24b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee490844e6c41ad97576ef04bbee24b, State: Initialized, Role: FOLLOWER
I20260928 23:53:44.922425 27214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ee490844e6c41ad97576ef04bbee24b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee490844e6c41ad97576ef04bbee24b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40569 } }
I20260928 23:53:44.922577 27214 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ee490844e6c41ad97576ef04bbee24b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:53:44.922669 27214 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ee490844e6c41ad97576ef04bbee24b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:53:44.922777 27214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ee490844e6c41ad97576ef04bbee24b [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:44.923818 27214 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ee490844e6c41ad97576ef04bbee24b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee490844e6c41ad97576ef04bbee24b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40569 } }
I20260928 23:53:44.923928 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:53:44.924216 27214 leader_election.cc:302] T 00000000000000000000000000000000 P 8ee490844e6c41ad97576ef04bbee24b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ee490844e6c41ad97576ef04bbee24b; no voters:
I20260928 23:53:44.924583 27214 leader_election.cc:288] T 00000000000000000000000000000000 P 8ee490844e6c41ad97576ef04bbee24b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:53:44.924721 27219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ee490844e6c41ad97576ef04bbee24b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:53:44.924986 27219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ee490844e6c41ad97576ef04bbee24b [term 1 LEADER]: Becoming Leader. State: Replica: 8ee490844e6c41ad97576ef04bbee24b, State: Running, Role: LEADER
I20260928 23:53:44.925359 27219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ee490844e6c41ad97576ef04bbee24b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee490844e6c41ad97576ef04bbee24b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40569 } }
I20260928 23:53:44.925890 27214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ee490844e6c41ad97576ef04bbee24b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:53:44.927150 27220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee490844e6c41ad97576ef04bbee24b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ee490844e6c41ad97576ef04bbee24b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee490844e6c41ad97576ef04bbee24b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40569 } } }
I20260928 23:53:44.927280 27220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee490844e6c41ad97576ef04bbee24b [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:44.927577 27221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee490844e6c41ad97576ef04bbee24b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ee490844e6c41ad97576ef04bbee24b. Latest consensus state: current_term: 1 leader_uuid: "8ee490844e6c41ad97576ef04bbee24b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee490844e6c41ad97576ef04bbee24b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40569 } } }
I20260928 23:53:44.927708 27221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee490844e6c41ad97576ef04bbee24b [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:44.931268 27217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:44.926372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39951 (local address 127.0.0.1:40569)
0928 23:53:44.926484 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:44.926488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:44.926666 (+ 178us) server_negotiation.cc:407] Connection header received
0928 23:53:44.926858 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:44.926862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:44.926910 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:44.926978 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:44.927361 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:44.928138 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:44.929271 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:44.929499 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:44.929579 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:44.930519 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:44.930523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:44.930526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:44.930539 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:44.930566 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:44.930578 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:44.930679 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:44.931115 (+ 436us) server_negotiation.cc:299] Negotiation successful
0928 23:53:44.931148 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 23:53:44.931488 27239 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ee490844e6c41ad97576ef04bbee24b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:53:44.931555 27239 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:53:44.931656 27233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:53:44.932463 27233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:53:44.937866 27233 catalog_manager.cc:1347] Generated new cluster ID: 089c4efc02bd43608934db650e13a135
I20260928 23:53:44.937924 27233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:53:44.949810 27233 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:53:44.950874 27233 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:53:44.962114 27233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ee490844e6c41ad97576ef04bbee24b: Generated new TSK 0
I20260928 23:53:44.962956 27233 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:53:44.973246 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 27146
W20260928 23:53:44.978638 27131 connection.cc:535] client connection to 127.0.0.1:40569 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40569): Connection reset by peer (error 104)
2026-09-28T23:53:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1394 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:53:45 dist-test-slave-p5bw krb5kdc[27245](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:53:45 dist-test-slave-p5bw krb5kdc[27245](info): set up 2 sockets
Sep 28 23:53:45 dist-test-slave-p5bw krb5kdc[27245](info): commencing operation
krb5kdc: starting...
W20260928 23:53:45.997655 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:53:46 dist-test-slave-p5bw krb5kdc[27245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:53:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:53:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:53:46.115247 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43377
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:53:46.221653 27258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:46.221915 27258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:46.221998 27258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:46.226274 27258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:53:46.226336 27258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:46.226356 27258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:46.226408 27258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:53:46.226436 27258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:53:46.231876 27258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43377
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:46.232988 27258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:46.233990 27258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:46.242180 27266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:46.242182 27264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:46.242184 27263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:46.242241 27258 server_base.cc:1034] running on GCE node
I20260928 23:53:46.243062 27258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:46.244154 27258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:46.245332 27258 hybrid_clock.cc:648] HybridClock initialized: now 1790639626245307 us; error 41 us; skew 500 ppm
Sep 28 23:53:46 dist-test-slave-p5bw krb5kdc[27245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:46.247932 27258 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:53:46.248780 27258 webserver.cc:460] Webserver started at http://127.0.0.1:33519/ using document root <none> and password file <none>
I20260928 23:53:46.249321 27258 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:46.249432 27258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:46.249645 27258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:46.251573 27258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "0bd3d96374624afc9b58bf48c967b2e4"
format_stamp: "Formatted at 2026-09-28 23:53:46 on dist-test-slave-p5bw"
server_key: "d92c82402b8a7d1a1cebb198cf14a070"
server_key_iv: "888684b0a2d141cd68553645df81d5a3"
server_key_version: "encryptionkey@0"
I20260928 23:53:46.252115 27258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "0bd3d96374624afc9b58bf48c967b2e4"
format_stamp: "Formatted at 2026-09-28 23:53:46 on dist-test-slave-p5bw"
server_key: "d92c82402b8a7d1a1cebb198cf14a070"
server_key_iv: "888684b0a2d141cd68553645df81d5a3"
server_key_version: "encryptionkey@0"
I20260928 23:53:46.255577 27258 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260928 23:53:46.257858 27273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:46.258972 27258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:53:46.259104 27258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "0bd3d96374624afc9b58bf48c967b2e4"
format_stamp: "Formatted at 2026-09-28 23:53:46 on dist-test-slave-p5bw"
server_key: "d92c82402b8a7d1a1cebb198cf14a070"
server_key_iv: "888684b0a2d141cd68553645df81d5a3"
server_key_version: "encryptionkey@0"
I20260928 23:53:46.259207 27258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:46.263470 27258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:46.266602 27258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:46.266788 27258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:46.273636 27258 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43377
I20260928 23:53:46.273662 27325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43377 every 8 connection(s)
I20260928 23:53:46.274519 27258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:53:46.277100 27326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:46.280994 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 27258
I20260928 23:53:46.281147 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:53:46.281453 6752 external_mini_cluster.cc:1428] Setting key f306a86a01a0573036c19bb2e53e8a5a
I20260928 23:53:46.283249 27326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bd3d96374624afc9b58bf48c967b2e4: Bootstrap starting.
I20260928 23:53:46.285729 27326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bd3d96374624afc9b58bf48c967b2e4: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:46.286375 27329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:46.282712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47565 (local address 127.0.0.1:43377)
0928 23:53:46.283092 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:46.283114 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:46.283149 (+ 35us) server_negotiation.cc:407] Connection header received
0928 23:53:46.283809 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:46.283826 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:46.284091 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:46.284475 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:46.285239 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:46.285242 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:46.285253 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:46.285261 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:46.285307 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:46.285324 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:46.285416 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:46.285535 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 23:53:46.285693 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":122,"threads_started":1}
I20260928 23:53:46.286444 27326 log.cc:824] T 00000000000000000000000000000000 P 0bd3d96374624afc9b58bf48c967b2e4: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:46.288233 27326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bd3d96374624afc9b58bf48c967b2e4: No bootstrap required, opened a new log
I20260928 23:53:46.289037 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:53:46.292165 27326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bd3d96374624afc9b58bf48c967b2e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bd3d96374624afc9b58bf48c967b2e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43377 } }
I20260928 23:53:46.292505 27326 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bd3d96374624afc9b58bf48c967b2e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:46.292506 27329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:46.290945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51417 (local address 127.0.0.1:43377)
0928 23:53:46.291048 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:46.291052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:46.291287 (+ 235us) server_negotiation.cc:407] Connection header received
0928 23:53:46.291392 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:46.291394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:46.291422 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:46.291483 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:46.292144 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:46.292146 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:46.292148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:46.292157 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:46.292178 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:46.292188 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:46.292250 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:46.292358 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 23:53:46.292385 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 23:53:46.292589 27326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bd3d96374624afc9b58bf48c967b2e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bd3d96374624afc9b58bf48c967b2e4, State: Initialized, Role: FOLLOWER
I20260928 23:53:46.293087 27326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bd3d96374624afc9b58bf48c967b2e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bd3d96374624afc9b58bf48c967b2e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43377 } }
I20260928 23:53:46.293241 27326 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bd3d96374624afc9b58bf48c967b2e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:53:46.293306 27326 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bd3d96374624afc9b58bf48c967b2e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:53:46.293403 27326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bd3d96374624afc9b58bf48c967b2e4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:46.294255 27326 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bd3d96374624afc9b58bf48c967b2e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bd3d96374624afc9b58bf48c967b2e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43377 } }
I20260928 23:53:46.294595 27326 leader_election.cc:302] T 00000000000000000000000000000000 P 0bd3d96374624afc9b58bf48c967b2e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bd3d96374624afc9b58bf48c967b2e4; no voters:
I20260928 23:53:46.294880 27326 leader_election.cc:288] T 00000000000000000000000000000000 P 0bd3d96374624afc9b58bf48c967b2e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:53:46.294978 27336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bd3d96374624afc9b58bf48c967b2e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:53:46.295228 27336 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bd3d96374624afc9b58bf48c967b2e4 [term 1 LEADER]: Becoming Leader. State: Replica: 0bd3d96374624afc9b58bf48c967b2e4, State: Running, Role: LEADER
I20260928 23:53:46.295554 27336 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bd3d96374624afc9b58bf48c967b2e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bd3d96374624afc9b58bf48c967b2e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43377 } }
I20260928 23:53:46.296074 27326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bd3d96374624afc9b58bf48c967b2e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:53:46.297288 27338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bd3d96374624afc9b58bf48c967b2e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bd3d96374624afc9b58bf48c967b2e4. Latest consensus state: current_term: 1 leader_uuid: "0bd3d96374624afc9b58bf48c967b2e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bd3d96374624afc9b58bf48c967b2e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43377 } } }
I20260928 23:53:46.297319 27337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bd3d96374624afc9b58bf48c967b2e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bd3d96374624afc9b58bf48c967b2e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bd3d96374624afc9b58bf48c967b2e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43377 } } }
I20260928 23:53:46.297407 27338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bd3d96374624afc9b58bf48c967b2e4 [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:46.297420 27337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bd3d96374624afc9b58bf48c967b2e4 [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:46.298380 27345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:53:46.300869 27345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:53:46.305863 27345 catalog_manager.cc:1347] Generated new cluster ID: 7a0ada63801745bba55795c2da3fac0a
I20260928 23:53:46.305924 27345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:53:46.317068 27345 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:53:46.317353 27345 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:53:46.321548 27345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bd3d96374624afc9b58bf48c967b2e4: Generated new TSK 0
I20260928 23:53:46.322180 27345 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:53:46.326619 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 27258
2026-09-28T23:53:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:53:46 dist-test-slave-p5bw krb5kdc[27357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:53:46 dist-test-slave-p5bw krb5kdc[27357](info): set up 2 sockets
Sep 28 23:53:46 dist-test-slave-p5bw krb5kdc[27357](info): commencing operation
krb5kdc: starting...
W20260928 23:53:47.361579 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:53:47 dist-test-slave-p5bw krb5kdc[27357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:53:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:53:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:53:47.480002 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40931
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:53:47.589274 27370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:47.589525 27370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:47.589566 27370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:47.593526 27370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:53:47.593580 27370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:47.593600 27370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:47.593616 27370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:53:47.593631 27370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:53:47.599354 27370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:47.600360 27370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:47.601323 27370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:47.609246 27376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:47.609232 27378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:47.609212 27375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:47.609390 27370 server_base.cc:1034] running on GCE node
I20260928 23:53:47.610260 27370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:47.611296 27370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:47.612519 27370 hybrid_clock.cc:648] HybridClock initialized: now 1790639627612497 us; error 37 us; skew 500 ppm
Sep 28 23:53:47 dist-test-slave-p5bw krb5kdc[27357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:47.615216 27370 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:53:47.616096 27370 webserver.cc:460] Webserver started at http://127.0.0.1:33477/ using document root <none> and password file <none>
I20260928 23:53:47.616640 27370 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:47.616730 27370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:47.616986 27370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:47.618917 27370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "998404dbc256417cac433e3c7864e249"
format_stamp: "Formatted at 2026-09-28 23:53:47 on dist-test-slave-p5bw"
server_key: "972d8f9f52c3c3182ae54a18bf01fdfa"
server_key_iv: "7dd2c90b8f3c9add317f19de6604a7f8"
server_key_version: "encryptionkey@0"
I20260928 23:53:47.619467 27370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "998404dbc256417cac433e3c7864e249"
format_stamp: "Formatted at 2026-09-28 23:53:47 on dist-test-slave-p5bw"
server_key: "972d8f9f52c3c3182ae54a18bf01fdfa"
server_key_iv: "7dd2c90b8f3c9add317f19de6604a7f8"
server_key_version: "encryptionkey@0"
I20260928 23:53:47.622978 27370 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 23:53:47.625371 27385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:47.626540 27370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:53:47.626677 27370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "998404dbc256417cac433e3c7864e249"
format_stamp: "Formatted at 2026-09-28 23:53:47 on dist-test-slave-p5bw"
server_key: "972d8f9f52c3c3182ae54a18bf01fdfa"
server_key_iv: "7dd2c90b8f3c9add317f19de6604a7f8"
server_key_version: "encryptionkey@0"
I20260928 23:53:47.626788 27370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:47.645852 27370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:47.649165 27370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:47.649365 27370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:47.656183 27370 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40931
I20260928 23:53:47.656206 27437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40931 every 8 connection(s)
I20260928 23:53:47.657155 27370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:53:47.659768 27438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:47.665151 27438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 998404dbc256417cac433e3c7864e249: Bootstrap starting.
I20260928 23:53:47.665875 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 27370
I20260928 23:53:47.666036 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:53:47.666355 6752 external_mini_cluster.cc:1428] Setting key bd07a5b578e9e93200cf6032952bd7d0
I20260928 23:53:47.668030 27438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 998404dbc256417cac433e3c7864e249: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:47.668730 27438 log.cc:824] T 00000000000000000000000000000000 P 998404dbc256417cac433e3c7864e249: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:47.670493 27438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 998404dbc256417cac433e3c7864e249: No bootstrap required, opened a new log
I20260928 23:53:47.673761 27438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 998404dbc256417cac433e3c7864e249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998404dbc256417cac433e3c7864e249" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40931 } }
I20260928 23:53:47.673993 27438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 998404dbc256417cac433e3c7864e249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:47.674073 27438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 998404dbc256417cac433e3c7864e249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 998404dbc256417cac433e3c7864e249, State: Initialized, Role: FOLLOWER
I20260928 23:53:47.674594 27438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 998404dbc256417cac433e3c7864e249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998404dbc256417cac433e3c7864e249" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40931 } }
Sep 28 23:53:47 dist-test-slave-p5bw krb5kdc[27357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 23:53:47.674757 27438 raft_consensus.cc:388] T 00000000000000000000000000000000 P 998404dbc256417cac433e3c7864e249 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:53:47.674876 27438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 998404dbc256417cac433e3c7864e249 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:53:47.674984 27438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 998404dbc256417cac433e3c7864e249 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:47.675820 27438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 998404dbc256417cac433e3c7864e249 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998404dbc256417cac433e3c7864e249" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40931 } }
I20260928 23:53:47.676190 27438 leader_election.cc:302] T 00000000000000000000000000000000 P 998404dbc256417cac433e3c7864e249 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 998404dbc256417cac433e3c7864e249; no voters:
I20260928 23:53:47.676537 27438 leader_election.cc:288] T 00000000000000000000000000000000 P 998404dbc256417cac433e3c7864e249 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:53:47.676644 27443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 998404dbc256417cac433e3c7864e249 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:53:47.676903 27443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 998404dbc256417cac433e3c7864e249 [term 1 LEADER]: Becoming Leader. State: Replica: 998404dbc256417cac433e3c7864e249, State: Running, Role: LEADER
I20260928 23:53:47.677254 27443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 998404dbc256417cac433e3c7864e249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998404dbc256417cac433e3c7864e249" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40931 } }
I20260928 23:53:47.677832 27438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 998404dbc256417cac433e3c7864e249 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:53:47.679961 27444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 998404dbc256417cac433e3c7864e249 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "998404dbc256417cac433e3c7864e249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998404dbc256417cac433e3c7864e249" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40931 } } }
I20260928 23:53:47.680385 27444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 998404dbc256417cac433e3c7864e249 [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:47.680063 27445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 998404dbc256417cac433e3c7864e249 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 998404dbc256417cac433e3c7864e249. Latest consensus state: current_term: 1 leader_uuid: "998404dbc256417cac433e3c7864e249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998404dbc256417cac433e3c7864e249" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40931 } } }
I20260928 23:53:47.680719 27445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 998404dbc256417cac433e3c7864e249 [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:47.681113 27441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:47.667523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40849 (local address 127.0.0.1:40931)
0928 23:53:47.667932 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:47.667946 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:47.667979 (+ 33us) server_negotiation.cc:407] Connection header received
0928 23:53:47.668656 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:47.668673 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:47.669079 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:47.669396 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:47.670195 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:47.671200 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:47.672038 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:47.672509 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:47.672677 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:47.675381 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:47.675386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:47.675403 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:53:47.675428 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:47.677479 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:47.677990 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:47.677995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:47.678000 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:47.678148 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:47.678458 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:47.678461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:47.678463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:47.678962 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:47.679159 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:47.679787 (+ 628us) server_negotiation.cc:299] Negotiation successful
0928 23:53:47.680096 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":127,"threads_started":1}
W20260928 23:53:47.682752 27457 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 998404dbc256417cac433e3c7864e249: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:53:47.682828 27457 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:53:47.683020 27459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:53:47.683907 27459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:53:47.683957 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:53:47.689793 27459 catalog_manager.cc:1347] Generated new cluster ID: 8988ff1f164a4f69aa206d798dc2089c
I20260928 23:53:47.689850 27459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:53:47.697806 27459 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:53:47.698973 27459 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:53:47.700877 27441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:47.685883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52373 (local address 127.0.0.1:40931)
0928 23:53:47.685988 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:47.685992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:47.686242 (+ 250us) server_negotiation.cc:407] Connection header received
0928 23:53:47.686372 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:47.686375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:47.686427 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:47.686501 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:47.687296 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:47.688048 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:47.688885 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:47.689220 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:47.689330 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:47.697551 (+ 8221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:47.697555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:47.697557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:53:47.697583 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:47.699112 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:47.699534 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:47.699539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:47.699541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:47.699623 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:47.699895 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:47.699898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:47.699900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:47.700066 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:47.700209 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:47.700438 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 23:53:47.700567 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:53:47.708252 27459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 998404dbc256417cac433e3c7864e249: Generated new TSK 0
I20260928 23:53:47.709002 27459 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:53:47.713176 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 27370
2026-09-28T23:53:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:53:47 dist-test-slave-p5bw krb5kdc[27469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:53:47 dist-test-slave-p5bw krb5kdc[27469](info): set up 2 sockets
Sep 28 23:53:47 dist-test-slave-p5bw krb5kdc[27469](info): commencing operation
krb5kdc: starting...
W20260928 23:53:48.761670 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:53:48 dist-test-slave-p5bw krb5kdc[27469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:53:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:53:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:53:48.878077 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37957
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:53:48.984216 27482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:48.984498 27482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:48.984570 27482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:48.988610 27482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:53:48.988691 27482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:48.988729 27482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:48.988783 27482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:53:48.988839 27482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:53:48.994077 27482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:48.995057 27482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:48.996359 27482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:49.005064 27487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:49.005112 27488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:49.005133 27490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:49.005369 27482 server_base.cc:1034] running on GCE node
I20260928 23:53:49.005965 27482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:49.007033 27482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:49.008198 27482 hybrid_clock.cc:648] HybridClock initialized: now 1790639629008182 us; error 25 us; skew 500 ppm
Sep 28 23:53:49 dist-test-slave-p5bw krb5kdc[27469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:49.016268 27482 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:53:49.017309 27482 webserver.cc:460] Webserver started at http://127.0.0.1:41835/ using document root <none> and password file <none>
I20260928 23:53:49.017913 27482 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:49.018007 27482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:49.018245 27482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:49.020340 27482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "97c93d9801884ee9a3d4d1d63124620b"
format_stamp: "Formatted at 2026-09-28 23:53:49 on dist-test-slave-p5bw"
server_key: "9bdca66fedaa0b2d413d892a5404070b"
server_key_iv: "71cae5e28181cb7d05af477910a5bfbb"
server_key_version: "encryptionkey@0"
I20260928 23:53:49.021004 27482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "97c93d9801884ee9a3d4d1d63124620b"
format_stamp: "Formatted at 2026-09-28 23:53:49 on dist-test-slave-p5bw"
server_key: "9bdca66fedaa0b2d413d892a5404070b"
server_key_iv: "71cae5e28181cb7d05af477910a5bfbb"
server_key_version: "encryptionkey@0"
I20260928 23:53:49.024667 27482 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:53:49.027045 27497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:49.028218 27482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:53:49.028365 27482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "97c93d9801884ee9a3d4d1d63124620b"
format_stamp: "Formatted at 2026-09-28 23:53:49 on dist-test-slave-p5bw"
server_key: "9bdca66fedaa0b2d413d892a5404070b"
server_key_iv: "71cae5e28181cb7d05af477910a5bfbb"
server_key_version: "encryptionkey@0"
I20260928 23:53:49.028496 27482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:49.053854 27482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:49.056713 27482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:49.056964 27482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:49.063915 27482 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37957
I20260928 23:53:49.063946 27549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37957 every 8 connection(s)
I20260928 23:53:49.064888 27482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:53:49.067520 27550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:49.072745 27550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97c93d9801884ee9a3d4d1d63124620b: Bootstrap starting.
I20260928 23:53:49.074445 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 27482
I20260928 23:53:49.074591 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:53:49.074873 6752 external_mini_cluster.cc:1428] Setting key b1f68c45c78021076b17a3007e2e2d21
I20260928 23:53:49.075384 27550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97c93d9801884ee9a3d4d1d63124620b: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:49.076215 27550 log.cc:824] T 00000000000000000000000000000000 P 97c93d9801884ee9a3d4d1d63124620b: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:49.078140 27550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97c93d9801884ee9a3d4d1d63124620b: No bootstrap required, opened a new log
I20260928 23:53:49.081509 27550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97c93d9801884ee9a3d4d1d63124620b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c93d9801884ee9a3d4d1d63124620b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } }
I20260928 23:53:49.081736 27550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97c93d9801884ee9a3d4d1d63124620b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:49.081815 27550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97c93d9801884ee9a3d4d1d63124620b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97c93d9801884ee9a3d4d1d63124620b, State: Initialized, Role: FOLLOWER
I20260928 23:53:49.082258 27550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97c93d9801884ee9a3d4d1d63124620b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c93d9801884ee9a3d4d1d63124620b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } }
I20260928 23:53:49.082404 27550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97c93d9801884ee9a3d4d1d63124620b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:53:49.082494 27550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97c93d9801884ee9a3d4d1d63124620b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:53:49.082599 27550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97c93d9801884ee9a3d4d1d63124620b [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:49.083648 27550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97c93d9801884ee9a3d4d1d63124620b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c93d9801884ee9a3d4d1d63124620b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } }
I20260928 23:53:49.083822 27553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:49.076066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57541 (local address 127.0.0.1:37957)
0928 23:53:49.076519 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:49.076534 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:49.076596 (+ 62us) server_negotiation.cc:407] Connection header received
0928 23:53:49.077395 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:49.077414 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:49.077740 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:49.078097 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:49.078512 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:49.079453 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:49.080365 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:49.080903 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:49.081064 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:49.081916 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:49.081923 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:49.081938 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:49.082330 (+ 392us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:49.082384 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:49.082437 (+ 53us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:49.082530 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:49.082667 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 23:53:49.082882 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":123,"threads_started":1}
I20260928 23:53:49.083992 27550 leader_election.cc:302] T 00000000000000000000000000000000 P 97c93d9801884ee9a3d4d1d63124620b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97c93d9801884ee9a3d4d1d63124620b; no voters:
I20260928 23:53:49.084299 27550 leader_election.cc:288] T 00000000000000000000000000000000 P 97c93d9801884ee9a3d4d1d63124620b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:53:49.084447 27555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97c93d9801884ee9a3d4d1d63124620b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:53:49.084707 27555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97c93d9801884ee9a3d4d1d63124620b [term 1 LEADER]: Becoming Leader. State: Replica: 97c93d9801884ee9a3d4d1d63124620b, State: Running, Role: LEADER
I20260928 23:53:49.085023 27555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97c93d9801884ee9a3d4d1d63124620b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c93d9801884ee9a3d4d1d63124620b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } }
I20260928 23:53:49.085522 27550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97c93d9801884ee9a3d4d1d63124620b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:53:49.086791 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:53:49.086886 27557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97c93d9801884ee9a3d4d1d63124620b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97c93d9801884ee9a3d4d1d63124620b. Latest consensus state: current_term: 1 leader_uuid: "97c93d9801884ee9a3d4d1d63124620b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c93d9801884ee9a3d4d1d63124620b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } } }
I20260928 23:53:49.086997 27557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97c93d9801884ee9a3d4d1d63124620b [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:49.087469 27566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:53:49.088750 27556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97c93d9801884ee9a3d4d1d63124620b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97c93d9801884ee9a3d4d1d63124620b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c93d9801884ee9a3d4d1d63124620b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } } }
I20260928 23:53:49.089217 27556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97c93d9801884ee9a3d4d1d63124620b [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:49.090679 27566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:53:49.094152 27553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:49.088997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56229 (local address 127.0.0.1:37957)
0928 23:53:49.089096 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:49.089103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:49.090020 (+ 917us) server_negotiation.cc:407] Connection header received
0928 23:53:49.090124 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:49.090127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:49.090171 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:49.090257 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:49.090558 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:49.091440 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:49.092557 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:49.092798 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:49.092937 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:49.093741 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:49.093745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:49.093748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:49.093759 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:49.093799 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:49.093809 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:49.093886 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:49.093973 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 23:53:49.094020 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 23:53:49.096220 27566 catalog_manager.cc:1347] Generated new cluster ID: bc76d96b89044dff861b5580fff5af5e
I20260928 23:53:49.096279 27566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:53:49.106788 27566 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:53:49.107638 27566 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:53:49.113354 27566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97c93d9801884ee9a3d4d1d63124620b: Generated new TSK 0
I20260928 23:53:49.113970 27566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:53:49.119191 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 27482
2026-09-28T23:53:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:53:49 dist-test-slave-p5bw krb5kdc[27581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:53:49 dist-test-slave-p5bw krb5kdc[27581](info): set up 2 sockets
Sep 28 23:53:49 dist-test-slave-p5bw krb5kdc[27581](info): commencing operation
krb5kdc: starting...
W20260928 23:53:50.153580 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:53:50 dist-test-slave-p5bw krb5kdc[27581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:53:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:53:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:53:50.271996 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:53:50.378561 27594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:50.378857 27594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:50.378942 27594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:50.382882 27594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:53:50.382970 27594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:50.383008 27594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:50.383044 27594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:53:50.383075 27594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:53:50.388170 27594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:50.389271 27594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:50.390283 27594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:50.398563 27602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:50.398715 27594 server_base.cc:1034] running on GCE node
W20260928 23:53:50.398566 27599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:50.398874 27600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:50.399396 27594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:50.400449 27594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:50.401633 27594 hybrid_clock.cc:648] HybridClock initialized: now 1790639630401591 us; error 58 us; skew 500 ppm
Sep 28 23:53:50 dist-test-slave-p5bw krb5kdc[27581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:50.404860 27594 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:53:50.405743 27594 webserver.cc:460] Webserver started at http://127.0.0.1:33141/ using document root <none> and password file <none>
I20260928 23:53:50.406291 27594 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:50.406383 27594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:50.406585 27594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:50.408564 27594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "2436a5e76c254f9fbc7bcd6ecef0728d"
format_stamp: "Formatted at 2026-09-28 23:53:50 on dist-test-slave-p5bw"
server_key: "76d35488585c0355d513ba3068327910"
server_key_iv: "55aff394c97875465ab9ca09441c4a00"
server_key_version: "encryptionkey@0"
I20260928 23:53:50.409140 27594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "2436a5e76c254f9fbc7bcd6ecef0728d"
format_stamp: "Formatted at 2026-09-28 23:53:50 on dist-test-slave-p5bw"
server_key: "76d35488585c0355d513ba3068327910"
server_key_iv: "55aff394c97875465ab9ca09441c4a00"
server_key_version: "encryptionkey@0"
I20260928 23:53:50.412715 27594 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 23:53:50.415123 27609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:50.416307 27594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:53:50.416445 27594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "2436a5e76c254f9fbc7bcd6ecef0728d"
format_stamp: "Formatted at 2026-09-28 23:53:50 on dist-test-slave-p5bw"
server_key: "76d35488585c0355d513ba3068327910"
server_key_iv: "55aff394c97875465ab9ca09441c4a00"
server_key_version: "encryptionkey@0"
I20260928 23:53:50.416555 27594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:50.421029 27594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:50.424161 27594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:50.424350 27594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:50.431440 27594 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46775
I20260928 23:53:50.431501 27661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46775 every 8 connection(s)
I20260928 23:53:50.432365 27594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:53:50.435001 27662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:50.437465 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 27594
I20260928 23:53:50.437628 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:53:50.437934 6752 external_mini_cluster.cc:1428] Setting key 5cf97ea27276297fff39901a4218533a
I20260928 23:53:50.440757 27662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2436a5e76c254f9fbc7bcd6ecef0728d: Bootstrap starting.
I20260928 23:53:50.442857 27665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:50.439291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54897 (local address 127.0.0.1:46775)
0928 23:53:50.439713 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:50.439726 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:50.439759 (+ 33us) server_negotiation.cc:407] Connection header received
0928 23:53:50.440348 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:50.440364 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:50.440630 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:50.440998 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:50.441758 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:50.441762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:50.441772 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:50.441782 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:50.441832 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:50.441857 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:50.441998 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:50.442066 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 23:53:50.442215 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":139,"threads_started":1}
I20260928 23:53:50.443110 27662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2436a5e76c254f9fbc7bcd6ecef0728d: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:50.443821 27662 log.cc:824] T 00000000000000000000000000000000 P 2436a5e76c254f9fbc7bcd6ecef0728d: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:50.445469 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:53:50.445744 27662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2436a5e76c254f9fbc7bcd6ecef0728d: No bootstrap required, opened a new log
I20260928 23:53:50.449084 27665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:50.447619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39959 (local address 127.0.0.1:46775)
0928 23:53:50.447723 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:50.447728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:50.447878 (+ 150us) server_negotiation.cc:407] Connection header received
0928 23:53:50.447992 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:50.447994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:50.448019 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:50.448081 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:50.448695 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:50.448698 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:50.448700 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:50.448709 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:50.448730 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:50.448739 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:50.448801 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:50.448940 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 23:53:50.448969 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 23:53:50.450335 27662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2436a5e76c254f9fbc7bcd6ecef0728d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2436a5e76c254f9fbc7bcd6ecef0728d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46775 } }
I20260928 23:53:50.450536 27662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2436a5e76c254f9fbc7bcd6ecef0728d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:50.450606 27662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2436a5e76c254f9fbc7bcd6ecef0728d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2436a5e76c254f9fbc7bcd6ecef0728d, State: Initialized, Role: FOLLOWER
I20260928 23:53:50.450981 27662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2436a5e76c254f9fbc7bcd6ecef0728d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2436a5e76c254f9fbc7bcd6ecef0728d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46775 } }
I20260928 23:53:50.451116 27662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2436a5e76c254f9fbc7bcd6ecef0728d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:53:50.451179 27662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2436a5e76c254f9fbc7bcd6ecef0728d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:53:50.451270 27662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2436a5e76c254f9fbc7bcd6ecef0728d [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:50.452205 27662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2436a5e76c254f9fbc7bcd6ecef0728d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2436a5e76c254f9fbc7bcd6ecef0728d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46775 } }
I20260928 23:53:50.452530 27662 leader_election.cc:302] T 00000000000000000000000000000000 P 2436a5e76c254f9fbc7bcd6ecef0728d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2436a5e76c254f9fbc7bcd6ecef0728d; no voters:
I20260928 23:53:50.452800 27662 leader_election.cc:288] T 00000000000000000000000000000000 P 2436a5e76c254f9fbc7bcd6ecef0728d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:53:50.452935 27672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2436a5e76c254f9fbc7bcd6ecef0728d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:53:50.453154 27672 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2436a5e76c254f9fbc7bcd6ecef0728d [term 1 LEADER]: Becoming Leader. State: Replica: 2436a5e76c254f9fbc7bcd6ecef0728d, State: Running, Role: LEADER
I20260928 23:53:50.453454 27672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2436a5e76c254f9fbc7bcd6ecef0728d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2436a5e76c254f9fbc7bcd6ecef0728d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46775 } }
I20260928 23:53:50.453946 27662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2436a5e76c254f9fbc7bcd6ecef0728d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:53:50.455006 27673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2436a5e76c254f9fbc7bcd6ecef0728d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2436a5e76c254f9fbc7bcd6ecef0728d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2436a5e76c254f9fbc7bcd6ecef0728d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46775 } } }
I20260928 23:53:50.455132 27673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2436a5e76c254f9fbc7bcd6ecef0728d [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:50.455448 27674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2436a5e76c254f9fbc7bcd6ecef0728d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2436a5e76c254f9fbc7bcd6ecef0728d. Latest consensus state: current_term: 1 leader_uuid: "2436a5e76c254f9fbc7bcd6ecef0728d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2436a5e76c254f9fbc7bcd6ecef0728d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46775 } } }
I20260928 23:53:50.455536 27674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2436a5e76c254f9fbc7bcd6ecef0728d [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:50.455521 27681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:53:50.458442 27681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:53:50.463318 27681 catalog_manager.cc:1347] Generated new cluster ID: bb87ea4fae19439a981af8ba0311299c
I20260928 23:53:50.463390 27681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:53:50.480042 27681 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:53:50.480319 27681 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:53:50.489459 27681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2436a5e76c254f9fbc7bcd6ecef0728d: Generated new TSK 0
I20260928 23:53:50.490095 27681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:53:50.497479 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 27594
2026-09-28T23:53:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1378 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:53:50 dist-test-slave-p5bw krb5kdc[27693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:53:50 dist-test-slave-p5bw krb5kdc[27693](info): set up 2 sockets
Sep 28 23:53:50 dist-test-slave-p5bw krb5kdc[27693](info): commencing operation
krb5kdc: starting...
W20260928 23:53:51.525681 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:53:51 dist-test-slave-p5bw krb5kdc[27693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:53:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:53:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:53:51.643822 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44593
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:53:51.753382 27706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:51.753688 27706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:51.753896 27706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:51.758036 27706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:53:51.758113 27706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:51.758160 27706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:51.758201 27706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:53:51.758256 27706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:53:51.763487 27706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:51.764550 27706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:51.765609 27706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:51.773940 27712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:51.773947 27711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:51.774052 27706 server_base.cc:1034] running on GCE node
W20260928 23:53:51.773916 27714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:51.774760 27706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:51.775833 27706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:51.777014 27706 hybrid_clock.cc:648] HybridClock initialized: now 1790639631776986 us; error 41 us; skew 500 ppm
Sep 28 23:53:51 dist-test-slave-p5bw krb5kdc[27693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:51.779632 27706 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:53:51.780443 27706 webserver.cc:460] Webserver started at http://127.0.0.1:45027/ using document root <none> and password file <none>
I20260928 23:53:51.781041 27706 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:51.781144 27706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:51.781363 27706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:51.783345 27706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "1880805f15af49d590edca324c992dae"
format_stamp: "Formatted at 2026-09-28 23:53:51 on dist-test-slave-p5bw"
server_key: "7cbfe445b530f81c05544ec8ab6849cb"
server_key_iv: "9cc618665f694d5412a4aaec340cc52e"
server_key_version: "encryptionkey@0"
I20260928 23:53:51.783895 27706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "1880805f15af49d590edca324c992dae"
format_stamp: "Formatted at 2026-09-28 23:53:51 on dist-test-slave-p5bw"
server_key: "7cbfe445b530f81c05544ec8ab6849cb"
server_key_iv: "9cc618665f694d5412a4aaec340cc52e"
server_key_version: "encryptionkey@0"
I20260928 23:53:51.787395 27706 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 23:53:51.789744 27721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:51.790898 27706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:53:51.791029 27706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "1880805f15af49d590edca324c992dae"
format_stamp: "Formatted at 2026-09-28 23:53:51 on dist-test-slave-p5bw"
server_key: "7cbfe445b530f81c05544ec8ab6849cb"
server_key_iv: "9cc618665f694d5412a4aaec340cc52e"
server_key_version: "encryptionkey@0"
I20260928 23:53:51.791134 27706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:51.822319 27706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:51.825251 27706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:51.825452 27706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:51.832293 27706 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44593
I20260928 23:53:51.832302 27773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44593 every 8 connection(s)
I20260928 23:53:51.833295 27706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:53:51.836021 27774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:51.839908 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 27706
I20260928 23:53:51.840058 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:53:51.840363 6752 external_mini_cluster.cc:1428] Setting key 5695ce6f9f1ad2362f7e64e2814263e1
I20260928 23:53:51.841658 27774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1880805f15af49d590edca324c992dae: Bootstrap starting.
I20260928 23:53:51.843974 27774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1880805f15af49d590edca324c992dae: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:51.844762 27774 log.cc:824] T 00000000000000000000000000000000 P 1880805f15af49d590edca324c992dae: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:51.846793 27774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1880805f15af49d590edca324c992dae: No bootstrap required, opened a new log
Sep 28 23:53:51 dist-test-slave-p5bw krb5kdc[27693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 23:53:51.851156 27774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1880805f15af49d590edca324c992dae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1880805f15af49d590edca324c992dae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44593 } }
I20260928 23:53:51.851388 27774 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1880805f15af49d590edca324c992dae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:51.851475 27774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1880805f15af49d590edca324c992dae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1880805f15af49d590edca324c992dae, State: Initialized, Role: FOLLOWER
I20260928 23:53:51.851948 27774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1880805f15af49d590edca324c992dae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1880805f15af49d590edca324c992dae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44593 } }
I20260928 23:53:51.852113 27774 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1880805f15af49d590edca324c992dae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:53:51.852219 27774 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1880805f15af49d590edca324c992dae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:53:51.852327 27774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1880805f15af49d590edca324c992dae [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:51.853351 27774 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1880805f15af49d590edca324c992dae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1880805f15af49d590edca324c992dae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44593 } }
I20260928 23:53:51.853721 27774 leader_election.cc:302] T 00000000000000000000000000000000 P 1880805f15af49d590edca324c992dae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1880805f15af49d590edca324c992dae; no voters:
I20260928 23:53:51.854053 27774 leader_election.cc:288] T 00000000000000000000000000000000 P 1880805f15af49d590edca324c992dae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:53:51.854148 27779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1880805f15af49d590edca324c992dae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:53:51.854357 27779 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1880805f15af49d590edca324c992dae [term 1 LEADER]: Becoming Leader. State: Replica: 1880805f15af49d590edca324c992dae, State: Running, Role: LEADER
I20260928 23:53:51.854717 27779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1880805f15af49d590edca324c992dae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1880805f15af49d590edca324c992dae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44593 } }
I20260928 23:53:51.854990 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:51.841687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.40:48795 (local address 127.0.0.1:44593)
0928 23:53:51.842110 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:51.842128 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:51.842171 (+ 43us) server_negotiation.cc:407] Connection header received
0928 23:53:51.842780 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:51.842795 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:51.843134 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:51.843421 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:51.844389 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:51.845446 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:51.846534 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:51.847032 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:51.847141 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:51.849867 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:51.849871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:51.849884 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:53:51.849909 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:51.851652 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:51.852231 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:51.852239 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:51.852246 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:51.852448 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:51.852794 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:51.852796 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:51.852797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:51.853208 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:51.853391 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:51.853685 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 23:53:51.853919 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":137,"threads_started":1}
I20260928 23:53:51.855278 27774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1880805f15af49d590edca324c992dae [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:53:51.855836 27779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1880805f15af49d590edca324c992dae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1880805f15af49d590edca324c992dae. Latest consensus state: current_term: 1 leader_uuid: "1880805f15af49d590edca324c992dae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1880805f15af49d590edca324c992dae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44593 } } }
I20260928 23:53:51.856520 27779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1880805f15af49d590edca324c992dae [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:51.856415 27780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1880805f15af49d590edca324c992dae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1880805f15af49d590edca324c992dae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1880805f15af49d590edca324c992dae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44593 } } }
I20260928 23:53:51.856796 27780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1880805f15af49d590edca324c992dae [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:51.858219 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 23:53:51.860620 27794 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1880805f15af49d590edca324c992dae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:53:51.860677 27794 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:53:51.860752 27788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:53:51.861716 27788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:53:51.866914 27788 catalog_manager.cc:1347] Generated new cluster ID: f9fc511a43914f8ebab0717ceeca4bfa
I20260928 23:53:51.867257 27788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:53:51.869590 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:51.860311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.40:50779 (local address 127.0.0.1:44593)
0928 23:53:51.860414 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:51.860418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:51.861116 (+ 698us) server_negotiation.cc:407] Connection header received
0928 23:53:51.861223 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:51.861226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:51.861272 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:51.861359 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:51.862134 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:51.862926 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:51.864032 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:51.864455 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:51.864579 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:51.866181 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:51.866184 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:51.866186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:53:51.866210 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:51.867757 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:51.868232 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:51.868236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:51.868238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:51.868318 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:51.868651 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:51.868654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:51.868656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:51.868880 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:51.869037 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:51.869303 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 23:53:51.869404 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:53:51.891263 27788 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:53:51.892397 27788 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:53:51.897377 27788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1880805f15af49d590edca324c992dae: Generated new TSK 0
I20260928 23:53:51.898149 27788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:53:51.901940 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 27706
W20260928 23:53:51.907183 27691 connection.cc:535] client connection to 127.0.0.1:44593 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44593): Connection reset by peer (error 104)
2026-09-28T23:53:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:53:51 dist-test-slave-p5bw krb5kdc[27805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:53:51 dist-test-slave-p5bw krb5kdc[27805](info): set up 2 sockets
Sep 28 23:53:51 dist-test-slave-p5bw krb5kdc[27805](info): commencing operation
krb5kdc: starting...
W20260928 23:53:52.921540 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:53:53 dist-test-slave-p5bw krb5kdc[27805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:53:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:53:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:53:53.041805 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45697
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:53:53.148552 27818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:53.148876 27818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:53.148988 27818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:53.152925 27818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:53:53.153014 27818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:53.153052 27818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:53.153088 27818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:53:53.153120 27818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:53:53.158299 27818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:53.159346 27818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:53.160356 27818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:53.168725 27826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:53.168713 27824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:53.168777 27818 server_base.cc:1034] running on GCE node
W20260928 23:53:53.168713 27823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:53.169503 27818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:53.170511 27818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:53.171712 27818 hybrid_clock.cc:648] HybridClock initialized: now 1790639633171698 us; error 24 us; skew 500 ppm
Sep 28 23:53:53 dist-test-slave-p5bw krb5kdc[27805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:53.174376 27818 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:53:53.175206 27818 webserver.cc:460] Webserver started at http://127.0.0.1:37339/ using document root <none> and password file <none>
I20260928 23:53:53.175711 27818 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:53.175768 27818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:53.175931 27818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:53.177894 27818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "5a737272cefc49dfbf3066bedcb5035e"
format_stamp: "Formatted at 2026-09-28 23:53:53 on dist-test-slave-p5bw"
server_key: "7bd4a280b5dca50ddcd7e26ee67abee0"
server_key_iv: "1a880872a4acef34e0115c3f6a450b4e"
server_key_version: "encryptionkey@0"
I20260928 23:53:53.178391 27818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "5a737272cefc49dfbf3066bedcb5035e"
format_stamp: "Formatted at 2026-09-28 23:53:53 on dist-test-slave-p5bw"
server_key: "7bd4a280b5dca50ddcd7e26ee67abee0"
server_key_iv: "1a880872a4acef34e0115c3f6a450b4e"
server_key_version: "encryptionkey@0"
I20260928 23:53:53.182008 27818 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260928 23:53:53.184476 27833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:53.185729 27818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 23:53:53.185825 27818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "5a737272cefc49dfbf3066bedcb5035e"
format_stamp: "Formatted at 2026-09-28 23:53:53 on dist-test-slave-p5bw"
server_key: "7bd4a280b5dca50ddcd7e26ee67abee0"
server_key_iv: "1a880872a4acef34e0115c3f6a450b4e"
server_key_version: "encryptionkey@0"
I20260928 23:53:53.185916 27818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:53.200052 27818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:53.203058 27818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:53.203217 27818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:53.210376 27818 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45697
I20260928 23:53:53.210398 27885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45697 every 8 connection(s)
I20260928 23:53:53.211268 27818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:53:53.213997 27886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:53.217887 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 27818
I20260928 23:53:53.218044 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:53:53.218354 6752 external_mini_cluster.cc:1428] Setting key 51fe88aa9ff68f27f6fdc844cc5094ca
I20260928 23:53:53.219873 27886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a737272cefc49dfbf3066bedcb5035e: Bootstrap starting.
I20260928 23:53:53.222175 27886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a737272cefc49dfbf3066bedcb5035e: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:53.222913 27886 log.cc:824] T 00000000000000000000000000000000 P 5a737272cefc49dfbf3066bedcb5035e: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:53.224673 27886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a737272cefc49dfbf3066bedcb5035e: No bootstrap required, opened a new log
I20260928 23:53:53.226977 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:53.219694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.40:51109 (local address 127.0.0.1:45697)
0928 23:53:53.220166 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:53.220181 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:53.220214 (+ 33us) server_negotiation.cc:407] Connection header received
0928 23:53:53.220896 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:53.220920 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:53.221241 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:53.221562 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:53.221964 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:53.222966 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:53.223896 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:53.224358 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:53.224531 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:53.225430 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:53.225435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:53.225447 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:53.225804 (+ 357us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:53.225854 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:53.225875 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:53.226022 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:53.226180 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 23:53:53.226309 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":152,"threads_started":1}
I20260928 23:53:53.228363 27886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a737272cefc49dfbf3066bedcb5035e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a737272cefc49dfbf3066bedcb5035e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45697 } }
I20260928 23:53:53.228636 27886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a737272cefc49dfbf3066bedcb5035e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:53.228696 27886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a737272cefc49dfbf3066bedcb5035e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a737272cefc49dfbf3066bedcb5035e, State: Initialized, Role: FOLLOWER
I20260928 23:53:53.229159 27886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a737272cefc49dfbf3066bedcb5035e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a737272cefc49dfbf3066bedcb5035e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45697 } }
I20260928 23:53:53.229334 27886 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a737272cefc49dfbf3066bedcb5035e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:53:53.229405 27886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a737272cefc49dfbf3066bedcb5035e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:53:53.229508 27886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a737272cefc49dfbf3066bedcb5035e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:53.229874 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:53:53.230501 27886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a737272cefc49dfbf3066bedcb5035e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a737272cefc49dfbf3066bedcb5035e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45697 } }
I20260928 23:53:53.230829 27886 leader_election.cc:302] T 00000000000000000000000000000000 P 5a737272cefc49dfbf3066bedcb5035e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a737272cefc49dfbf3066bedcb5035e; no voters:
I20260928 23:53:53.231161 27886 leader_election.cc:288] T 00000000000000000000000000000000 P 5a737272cefc49dfbf3066bedcb5035e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:53:53.231251 27894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a737272cefc49dfbf3066bedcb5035e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:53:53.231582 27894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a737272cefc49dfbf3066bedcb5035e [term 1 LEADER]: Becoming Leader. State: Replica: 5a737272cefc49dfbf3066bedcb5035e, State: Running, Role: LEADER
I20260928 23:53:53.231968 27894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a737272cefc49dfbf3066bedcb5035e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a737272cefc49dfbf3066bedcb5035e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45697 } }
I20260928 23:53:53.232684 27886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a737272cefc49dfbf3066bedcb5035e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:53:53.233788 27896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a737272cefc49dfbf3066bedcb5035e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a737272cefc49dfbf3066bedcb5035e. Latest consensus state: current_term: 1 leader_uuid: "5a737272cefc49dfbf3066bedcb5035e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a737272cefc49dfbf3066bedcb5035e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45697 } } }
I20260928 23:53:53.233922 27896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a737272cefc49dfbf3066bedcb5035e [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:53.234194 27897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a737272cefc49dfbf3066bedcb5035e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a737272cefc49dfbf3066bedcb5035e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a737272cefc49dfbf3066bedcb5035e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45697 } } }
I20260928 23:53:53.234324 27897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a737272cefc49dfbf3066bedcb5035e [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:53.236992 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:53.232013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.40:52917 (local address 127.0.0.1:45697)
0928 23:53:53.232104 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:53.232108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:53.232450 (+ 342us) server_negotiation.cc:407] Connection header received
0928 23:53:53.232552 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:53.232555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:53.232601 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:53.232678 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:53.233163 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:53.234001 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:53.235184 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:53.235551 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:53.235632 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:53.236495 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:53.236498 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:53.236501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:53.236512 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:53.236537 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:53.236549 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:53.236638 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:53.236804 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 23:53:53.236859 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":26}
W20260928 23:53:53.237823 27911 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a737272cefc49dfbf3066bedcb5035e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:53:53.237905 27911 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:53:53.237967 27905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:53:53.238631 27905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:53:53.244493 27905 catalog_manager.cc:1347] Generated new cluster ID: 4b3ace74d4614263a10d3ac8687be72d
I20260928 23:53:53.244560 27905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:53:53.272935 27905 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:53:53.273910 27905 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:53:53.281064 27905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a737272cefc49dfbf3066bedcb5035e: Generated new TSK 0
I20260928 23:53:53.281790 27905 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:53:53.294697 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 27818
W20260928 23:53:53.300175 27803 connection.cc:535] client connection to 127.0.0.1:45697 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45697): Connection reset by peer (error 104)
2026-09-28T23:53:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:53:53 dist-test-slave-p5bw krb5kdc[27917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:53:53 dist-test-slave-p5bw krb5kdc[27917](info): set up 2 sockets
Sep 28 23:53:53 dist-test-slave-p5bw krb5kdc[27917](info): commencing operation
krb5kdc: starting...
W20260928 23:53:54.337723 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:53:54 dist-test-slave-p5bw krb5kdc[27917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:53:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:53:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:53:54.456116 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33675
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:53:54.562144 27930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:54.562407 27930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:54.562450 27930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:54.566497 27930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:53:54.566550 27930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:54.566570 27930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:54.566588 27930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:53:54.566639 27930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:53:54.571715 27930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:54.572763 27930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:54.573755 27930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:54.581996 27935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:54.582005 27938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:54.582134 27930 server_base.cc:1034] running on GCE node
W20260928 23:53:54.582008 27936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:54.582824 27930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:54.583855 27930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:54.585039 27930 hybrid_clock.cc:648] HybridClock initialized: now 1790639634585026 us; error 23 us; skew 500 ppm
Sep 28 23:53:54 dist-test-slave-p5bw krb5kdc[27917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:54.592134 27930 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:53:54.593032 27930 webserver.cc:460] Webserver started at http://127.0.0.1:45507/ using document root <none> and password file <none>
I20260928 23:53:54.593573 27930 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:54.593662 27930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:54.593891 27930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:54.595824 27930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "61c43ad8150b4109b1307044f02bdcd7"
format_stamp: "Formatted at 2026-09-28 23:53:54 on dist-test-slave-p5bw"
server_key: "d2cb4fe753fb996a7eae220e971f76e5"
server_key_iv: "d6bdeec97032f05a1d12fb9f209d446e"
server_key_version: "encryptionkey@0"
I20260928 23:53:54.596380 27930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "61c43ad8150b4109b1307044f02bdcd7"
format_stamp: "Formatted at 2026-09-28 23:53:54 on dist-test-slave-p5bw"
server_key: "d2cb4fe753fb996a7eae220e971f76e5"
server_key_iv: "d6bdeec97032f05a1d12fb9f209d446e"
server_key_version: "encryptionkey@0"
I20260928 23:53:54.599900 27930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 23:53:54.602324 27945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:54.603549 27930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:53:54.603683 27930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "61c43ad8150b4109b1307044f02bdcd7"
format_stamp: "Formatted at 2026-09-28 23:53:54 on dist-test-slave-p5bw"
server_key: "d2cb4fe753fb996a7eae220e971f76e5"
server_key_iv: "d6bdeec97032f05a1d12fb9f209d446e"
server_key_version: "encryptionkey@0"
I20260928 23:53:54.603793 27930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:54.608228 27930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:54.610951 27930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:54.611136 27930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:54.618008 27930 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33675
I20260928 23:53:54.618033 27997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33675 every 8 connection(s)
I20260928 23:53:54.618887 27930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:53:54.621459 27998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:54.621650 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 27930
I20260928 23:53:54.621778 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:53:54.622084 6752 external_mini_cluster.cc:1428] Setting key f8e165cd79d1b34054840824bd355ccf
I20260928 23:53:54.627547 28001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:54.623458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.40:34319 (local address 127.0.0.1:33675)
0928 23:53:54.623860 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:54.623877 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:54.623927 (+ 50us) server_negotiation.cc:407] Connection header received
0928 23:53:54.624682 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:54.624706 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:54.625096 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:54.625462 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:54.626187 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:54.626193 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:54.626206 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:54.626217 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:54.626278 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:54.626304 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:54.626444 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:54.626509 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 23:53:54.626663 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":104,"threads_started":1}
I20260928 23:53:54.628455 27998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61c43ad8150b4109b1307044f02bdcd7: Bootstrap starting.
I20260928 23:53:54.630294 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:53:54.630847 27998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61c43ad8150b4109b1307044f02bdcd7: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:54.631577 27998 log.cc:824] T 00000000000000000000000000000000 P 61c43ad8150b4109b1307044f02bdcd7: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:54.633719 27998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61c43ad8150b4109b1307044f02bdcd7: No bootstrap required, opened a new log
I20260928 23:53:54.633986 28001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:54.632382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.40:33581 (local address 127.0.0.1:33675)
0928 23:53:54.632504 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:54.632508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:54.632687 (+ 179us) server_negotiation.cc:407] Connection header received
0928 23:53:54.632762 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:54.632765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:54.632798 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:54.632912 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:54.633600 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:54.633603 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:54.633605 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:54.633616 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:54.633641 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:54.633653 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:54.633743 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:54.633839 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 23:53:54.633884 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:53:54.637311 27998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61c43ad8150b4109b1307044f02bdcd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c43ad8150b4109b1307044f02bdcd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33675 } }
I20260928 23:53:54.637568 27998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61c43ad8150b4109b1307044f02bdcd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:54.637634 27998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61c43ad8150b4109b1307044f02bdcd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c43ad8150b4109b1307044f02bdcd7, State: Initialized, Role: FOLLOWER
I20260928 23:53:54.638073 27998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61c43ad8150b4109b1307044f02bdcd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c43ad8150b4109b1307044f02bdcd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33675 } }
I20260928 23:53:54.638198 27998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61c43ad8150b4109b1307044f02bdcd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:53:54.638305 27998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61c43ad8150b4109b1307044f02bdcd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:53:54.638424 27998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61c43ad8150b4109b1307044f02bdcd7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:54.639333 27998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61c43ad8150b4109b1307044f02bdcd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c43ad8150b4109b1307044f02bdcd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33675 } }
I20260928 23:53:54.639667 27998 leader_election.cc:302] T 00000000000000000000000000000000 P 61c43ad8150b4109b1307044f02bdcd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61c43ad8150b4109b1307044f02bdcd7; no voters:
I20260928 23:53:54.639971 27998 leader_election.cc:288] T 00000000000000000000000000000000 P 61c43ad8150b4109b1307044f02bdcd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:53:54.640096 28008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61c43ad8150b4109b1307044f02bdcd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:53:54.640321 28008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61c43ad8150b4109b1307044f02bdcd7 [term 1 LEADER]: Becoming Leader. State: Replica: 61c43ad8150b4109b1307044f02bdcd7, State: Running, Role: LEADER
I20260928 23:53:54.640640 28008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61c43ad8150b4109b1307044f02bdcd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c43ad8150b4109b1307044f02bdcd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33675 } }
I20260928 23:53:54.641175 27998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61c43ad8150b4109b1307044f02bdcd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:53:54.642290 28010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c43ad8150b4109b1307044f02bdcd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61c43ad8150b4109b1307044f02bdcd7. Latest consensus state: current_term: 1 leader_uuid: "61c43ad8150b4109b1307044f02bdcd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c43ad8150b4109b1307044f02bdcd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33675 } } }
I20260928 23:53:54.642431 28010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c43ad8150b4109b1307044f02bdcd7 [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:54.642797 28017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:53:54.642772 28009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c43ad8150b4109b1307044f02bdcd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61c43ad8150b4109b1307044f02bdcd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c43ad8150b4109b1307044f02bdcd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33675 } } }
I20260928 23:53:54.643005 28009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c43ad8150b4109b1307044f02bdcd7 [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:54.645982 28017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:53:54.651446 28017 catalog_manager.cc:1347] Generated new cluster ID: cd0a13343b0745c89f23d1905186010e
I20260928 23:53:54.651521 28017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:53:54.669245 28017 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:53:54.669571 28017 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:53:54.676928 28017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61c43ad8150b4109b1307044f02bdcd7: Generated new TSK 0
I20260928 23:53:54.677593 28017 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:53:54.687736 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 27930
2026-09-28T23:53:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:53:54 dist-test-slave-p5bw krb5kdc[28029](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:53:54 dist-test-slave-p5bw krb5kdc[28029](info): set up 2 sockets
Sep 28 23:53:54 dist-test-slave-p5bw krb5kdc[28029](info): commencing operation
krb5kdc: starting...
W20260928 23:53:55.717711 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:53:55 dist-test-slave-p5bw krb5kdc[28029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:53:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:53:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:53:55.836737 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43673
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:53:55.947404 28042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:55.947705 28042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:55.947791 28042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:55.951812 28042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:53:55.951901 28042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:55.951943 28042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:55.952004 28042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:53:55.952044 28042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:53:55.957248 28042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43673
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:55.958290 28042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:55.959288 28042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:55.967633 28050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:55.967640 28048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:55.967648 28047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:55.967918 28042 server_base.cc:1034] running on GCE node
I20260928 23:53:55.968420 28042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:55.969558 28042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:55.970738 28042 hybrid_clock.cc:648] HybridClock initialized: now 1790639635970714 us; error 35 us; skew 500 ppm
Sep 28 23:53:55 dist-test-slave-p5bw krb5kdc[28029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:55.973438 28042 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:53:55.974249 28042 webserver.cc:460] Webserver started at http://127.0.0.1:43077/ using document root <none> and password file <none>
I20260928 23:53:55.974731 28042 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:55.974785 28042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:55.974937 28042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:55.976868 28042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "81bccdbad323484b8f2b3201743c8930"
format_stamp: "Formatted at 2026-09-28 23:53:55 on dist-test-slave-p5bw"
server_key: "0101f67b22aa624c11069b89b5073bfc"
server_key_iv: "bc3d793a019b012de2e9dc1febc23fc1"
server_key_version: "encryptionkey@0"
I20260928 23:53:55.977373 28042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "81bccdbad323484b8f2b3201743c8930"
format_stamp: "Formatted at 2026-09-28 23:53:55 on dist-test-slave-p5bw"
server_key: "0101f67b22aa624c11069b89b5073bfc"
server_key_iv: "bc3d793a019b012de2e9dc1febc23fc1"
server_key_version: "encryptionkey@0"
I20260928 23:53:55.980700 28042 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260928 23:53:55.982954 28057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:55.984107 28042 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:53:55.984238 28042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "81bccdbad323484b8f2b3201743c8930"
format_stamp: "Formatted at 2026-09-28 23:53:55 on dist-test-slave-p5bw"
server_key: "0101f67b22aa624c11069b89b5073bfc"
server_key_iv: "bc3d793a019b012de2e9dc1febc23fc1"
server_key_version: "encryptionkey@0"
I20260928 23:53:55.984345 28042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:56.008291 28042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:56.011093 28042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:56.011291 28042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:56.018162 28042 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43673
I20260928 23:53:56.018186 28109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43673 every 8 connection(s)
I20260928 23:53:56.019071 28042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:53:56.021698 28110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:56.022871 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 28042
I20260928 23:53:56.023037 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:53:56.023399 6752 external_mini_cluster.cc:1428] Setting key 2b2bdc51088048663b2cb1a39f2d11d6
I20260928 23:53:56.029057 28110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81bccdbad323484b8f2b3201743c8930: Bootstrap starting.
I20260928 23:53:56.031572 28110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81bccdbad323484b8f2b3201743c8930: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:56.032361 28110 log.cc:824] T 00000000000000000000000000000000 P 81bccdbad323484b8f2b3201743c8930: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:53:56 dist-test-slave-p5bw krb5kdc[28029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 23:53:56.034296 28110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81bccdbad323484b8f2b3201743c8930: No bootstrap required, opened a new log
I20260928 23:53:56.039060 28110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81bccdbad323484b8f2b3201743c8930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81bccdbad323484b8f2b3201743c8930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43673 } }
I20260928 23:53:56.039309 28110 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81bccdbad323484b8f2b3201743c8930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:56.039366 28110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81bccdbad323484b8f2b3201743c8930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81bccdbad323484b8f2b3201743c8930, State: Initialized, Role: FOLLOWER
I20260928 23:53:56.039585 28113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:56.025823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.40:47501 (local address 127.0.0.1:43673)
0928 23:53:56.026243 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:56.026262 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:56.026303 (+ 41us) server_negotiation.cc:407] Connection header received
0928 23:53:56.027165 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:56.027190 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:56.027641 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:56.028013 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:56.028668 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:56.030096 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:56.030955 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:56.031631 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:56.031795 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:56.034450 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:56.034456 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:56.034472 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:53:56.034501 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:56.036448 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:56.037023 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:56.037029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:56.037034 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:56.037169 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:56.037490 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:56.037493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:56.037495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:56.037992 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:56.038212 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:56.038434 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 23:53:56.038653 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":97,"threads_started":1}
I20260928 23:53:56.039829 28110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81bccdbad323484b8f2b3201743c8930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81bccdbad323484b8f2b3201743c8930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43673 } }
I20260928 23:53:56.039979 28110 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81bccdbad323484b8f2b3201743c8930 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:53:56.040043 28110 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81bccdbad323484b8f2b3201743c8930 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:53:56.040134 28110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81bccdbad323484b8f2b3201743c8930 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:56.041170 28110 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81bccdbad323484b8f2b3201743c8930 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81bccdbad323484b8f2b3201743c8930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43673 } }
I20260928 23:53:56.041543 28110 leader_election.cc:302] T 00000000000000000000000000000000 P 81bccdbad323484b8f2b3201743c8930 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81bccdbad323484b8f2b3201743c8930; no voters:
I20260928 23:53:56.041862 28110 leader_election.cc:288] T 00000000000000000000000000000000 P 81bccdbad323484b8f2b3201743c8930 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:53:56.041965 28115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81bccdbad323484b8f2b3201743c8930 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:53:56.042148 28115 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81bccdbad323484b8f2b3201743c8930 [term 1 LEADER]: Becoming Leader. State: Replica: 81bccdbad323484b8f2b3201743c8930, State: Running, Role: LEADER
I20260928 23:53:56.042600 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:53:56.043164 28110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81bccdbad323484b8f2b3201743c8930 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:53:56.042478 28115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81bccdbad323484b8f2b3201743c8930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81bccdbad323484b8f2b3201743c8930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43673 } }
I20260928 23:53:56.046299 28117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81bccdbad323484b8f2b3201743c8930 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81bccdbad323484b8f2b3201743c8930. Latest consensus state: current_term: 1 leader_uuid: "81bccdbad323484b8f2b3201743c8930" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81bccdbad323484b8f2b3201743c8930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43673 } } }
I20260928 23:53:56.046299 28116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81bccdbad323484b8f2b3201743c8930 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81bccdbad323484b8f2b3201743c8930" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81bccdbad323484b8f2b3201743c8930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43673 } } }
I20260928 23:53:56.046452 28116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81bccdbad323484b8f2b3201743c8930 [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:56.046452 28117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81bccdbad323484b8f2b3201743c8930 [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:56.046854 28135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:53:56.049932 28135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:53:56.054227 28113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:56.044870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.40:37779 (local address 127.0.0.1:43673)
0928 23:53:56.044975 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:56.044982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:56.045303 (+ 321us) server_negotiation.cc:407] Connection header received
0928 23:53:56.045445 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:56.045449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:56.045498 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:56.045595 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:56.047100 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:56.047707 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:56.048763 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:56.049125 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:56.049230 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:56.050876 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:56.050880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:56.050882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:53:56.050906 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:56.052442 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:56.052951 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:56.052954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:56.052956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:56.053061 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:53:56.053399 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:53:56.053402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:53:56.053404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:53:56.053571 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:56.053718 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:56.053991 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 23:53:56.054082 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 23:53:56.056365 28135 catalog_manager.cc:1347] Generated new cluster ID: 8d3c4f8828a946fca3f95dfae83f904d
I20260928 23:53:56.056427 28135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:53:56.071837 28135 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:53:56.072728 28135 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:53:56.080142 28135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81bccdbad323484b8f2b3201743c8930: Generated new TSK 0
I20260928 23:53:56.080937 28135 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:53:56.089718 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 28042
W20260928 23:53:56.095184 28027 connection.cc:535] client connection to 127.0.0.1:43673 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43673): Connection reset by peer (error 104)
2026-09-28T23:53:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:53:56 dist-test-slave-p5bw krb5kdc[28141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:53:56 dist-test-slave-p5bw krb5kdc[28141](info): set up 2 sockets
Sep 28 23:53:56 dist-test-slave-p5bw krb5kdc[28141](info): commencing operation
krb5kdc: starting...
W20260928 23:53:57.121622 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:53:57 dist-test-slave-p5bw krb5kdc[28141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:53:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:53:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:53:57.237866 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43019
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:53:57.345769 28154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:57.346063 28154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:57.346135 28154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:57.350129 28154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:53:57.350211 28154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:57.350248 28154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:57.350308 28154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:53:57.350343 28154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:53:57.355439 28154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43019
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:57.356446 28154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:57.357484 28154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:57.365674 28162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:57.365672 28159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:57.365681 28160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:57.365898 28154 server_base.cc:1034] running on GCE node
I20260928 23:53:57.366513 28154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:57.367476 28154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:57.368640 28154 hybrid_clock.cc:648] HybridClock initialized: now 1790639637368621 us; error 30 us; skew 500 ppm
Sep 28 23:53:57 dist-test-slave-p5bw krb5kdc[28141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:57.371307 28154 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:53:57.372102 28154 webserver.cc:460] Webserver started at http://127.0.0.1:38033/ using document root <none> and password file <none>
I20260928 23:53:57.372623 28154 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:57.372681 28154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:57.372954 28154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:57.374855 28154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "1e8cfc5120ce4a7aa029ffb215d67d71"
format_stamp: "Formatted at 2026-09-28 23:53:57 on dist-test-slave-p5bw"
server_key: "bf2d68f2c9deca9081168aa9811c32c3"
server_key_iv: "8f4a9e48844689b8c758901e871fdf05"
server_key_version: "encryptionkey@0"
I20260928 23:53:57.375414 28154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "1e8cfc5120ce4a7aa029ffb215d67d71"
format_stamp: "Formatted at 2026-09-28 23:53:57 on dist-test-slave-p5bw"
server_key: "bf2d68f2c9deca9081168aa9811c32c3"
server_key_iv: "8f4a9e48844689b8c758901e871fdf05"
server_key_version: "encryptionkey@0"
I20260928 23:53:57.378890 28154 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 23:53:57.381110 28169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:57.382292 28154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 23:53:57.382449 28154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "1e8cfc5120ce4a7aa029ffb215d67d71"
format_stamp: "Formatted at 2026-09-28 23:53:57 on dist-test-slave-p5bw"
server_key: "bf2d68f2c9deca9081168aa9811c32c3"
server_key_iv: "8f4a9e48844689b8c758901e871fdf05"
server_key_version: "encryptionkey@0"
I20260928 23:53:57.382558 28154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:57.401242 28154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:57.404083 28154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:57.404284 28154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:57.411221 28154 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43019
I20260928 23:53:57.411240 28221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43019 every 8 connection(s)
I20260928 23:53:57.412161 28154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:53:57.413582 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 28154
I20260928 23:53:57.413722 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:53:57.414013 6752 external_mini_cluster.cc:1428] Setting key 950742d8e3f4e0baab3ca083ab3618e9
I20260928 23:53:57.415951 28222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:57.421532 28222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e8cfc5120ce4a7aa029ffb215d67d71: Bootstrap starting.
I20260928 23:53:57.422649 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:57.415249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.40:57799 (local address 127.0.0.1:43019)
0928 23:53:57.415637 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:57.415651 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:57.415684 (+ 33us) server_negotiation.cc:407] Connection header received
0928 23:53:57.416380 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:57.416398 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:57.416737 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:57.417129 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:57.417519 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:57.418478 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:57.419386 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:57.419868 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:57.420054 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:57.421038 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:57.421043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:57.421055 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:57.421428 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:57.421480 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:57.421501 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:57.421639 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:57.421819 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 23:53:57.421963 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":121,"threads_started":1}
I20260928 23:53:57.424000 28222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e8cfc5120ce4a7aa029ffb215d67d71: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:57.424883 28222 log.cc:824] T 00000000000000000000000000000000 P 1e8cfc5120ce4a7aa029ffb215d67d71: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:57.425649 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:53:57.426836 28222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e8cfc5120ce4a7aa029ffb215d67d71: No bootstrap required, opened a new log
I20260928 23:53:57.430605 28222 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e8cfc5120ce4a7aa029ffb215d67d71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e8cfc5120ce4a7aa029ffb215d67d71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43019 } }
I20260928 23:53:57.430846 28222 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e8cfc5120ce4a7aa029ffb215d67d71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:57.430917 28222 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e8cfc5120ce4a7aa029ffb215d67d71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e8cfc5120ce4a7aa029ffb215d67d71, State: Initialized, Role: FOLLOWER
I20260928 23:53:57.431435 28222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e8cfc5120ce4a7aa029ffb215d67d71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e8cfc5120ce4a7aa029ffb215d67d71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43019 } }
I20260928 23:53:57.431609 28222 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e8cfc5120ce4a7aa029ffb215d67d71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:53:57.431689 28222 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e8cfc5120ce4a7aa029ffb215d67d71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:53:57.431802 28222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e8cfc5120ce4a7aa029ffb215d67d71 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:57.432399 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:57.427636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.40:37151 (local address 127.0.0.1:43019)
0928 23:53:57.427760 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:57.427764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:57.428025 (+ 261us) server_negotiation.cc:407] Connection header received
0928 23:53:57.428134 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:57.428136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:57.428186 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:57.428250 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:57.428530 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:57.429454 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:57.430601 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:53:57.430877 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:53:57.431011 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:53:57.431920 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:57.431926 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:57.431933 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:57.431945 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:57.431970 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:57.431980 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:57.432081 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:57.432227 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 23:53:57.432262 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:53:57.432997 28222 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e8cfc5120ce4a7aa029ffb215d67d71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e8cfc5120ce4a7aa029ffb215d67d71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43019 } }
I20260928 23:53:57.433399 28222 leader_election.cc:302] T 00000000000000000000000000000000 P 1e8cfc5120ce4a7aa029ffb215d67d71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e8cfc5120ce4a7aa029ffb215d67d71; no voters:
I20260928 23:53:57.433729 28222 leader_election.cc:288] T 00000000000000000000000000000000 P 1e8cfc5120ce4a7aa029ffb215d67d71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:53:57.433847 28232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e8cfc5120ce4a7aa029ffb215d67d71 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:53:57.434063 28232 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e8cfc5120ce4a7aa029ffb215d67d71 [term 1 LEADER]: Becoming Leader. State: Replica: 1e8cfc5120ce4a7aa029ffb215d67d71, State: Running, Role: LEADER
I20260928 23:53:57.434419 28232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e8cfc5120ce4a7aa029ffb215d67d71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e8cfc5120ce4a7aa029ffb215d67d71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43019 } }
I20260928 23:53:57.435026 28222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e8cfc5120ce4a7aa029ffb215d67d71 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:53:57.436218 28233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e8cfc5120ce4a7aa029ffb215d67d71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e8cfc5120ce4a7aa029ffb215d67d71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e8cfc5120ce4a7aa029ffb215d67d71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43019 } } }
I20260928 23:53:57.436235 28234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e8cfc5120ce4a7aa029ffb215d67d71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e8cfc5120ce4a7aa029ffb215d67d71. Latest consensus state: current_term: 1 leader_uuid: "1e8cfc5120ce4a7aa029ffb215d67d71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e8cfc5120ce4a7aa029ffb215d67d71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43019 } } }
I20260928 23:53:57.436367 28233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e8cfc5120ce4a7aa029ffb215d67d71 [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:57.436391 28234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e8cfc5120ce4a7aa029ffb215d67d71 [sys.catalog]: This master's current role is: LEADER
W20260928 23:53:57.439658 28247 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e8cfc5120ce4a7aa029ffb215d67d71: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:53:57.439755 28247 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:53:57.439829 28242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:53:57.440487 28242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:53:57.445828 28242 catalog_manager.cc:1347] Generated new cluster ID: ba003db411e24be7a029a89bcff204b1
I20260928 23:53:57.445889 28242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:53:57.460469 28242 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:53:57.461390 28242 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:53:57.469878 28242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e8cfc5120ce4a7aa029ffb215d67d71: Generated new TSK 0
I20260928 23:53:57.470518 28242 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:53:57.476928 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 28154
W20260928 23:53:57.482438 28139 connection.cc:535] client connection to 127.0.0.1:43019 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43019): Connection reset by peer (error 104)
2026-09-28T23:53:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:53:57 dist-test-slave-p5bw krb5kdc[28253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:53:57 dist-test-slave-p5bw krb5kdc[28253](info): set up 2 sockets
Sep 28 23:53:57 dist-test-slave-p5bw krb5kdc[28253](info): commencing operation
krb5kdc: starting...
W20260928 23:53:58.521646 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:53:58 dist-test-slave-p5bw krb5kdc[28253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:53:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:53:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:53:58.645565 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38665
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:53:58.752038 28266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:53:58.752274 28266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:53:58.752314 28266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:53:58.756202 28266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:53:58.756255 28266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:53:58.756274 28266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:53:58.756290 28266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:53:58.756306 28266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:53:58.761437 28266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38665
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:53:58.762356 28266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:53:58.763322 28266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:53:58.771508 28271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:58.771515 28274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:53:58.771541 28272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:53:58.771550 28266 server_base.cc:1034] running on GCE node
I20260928 23:53:58.772231 28266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:53:58.773307 28266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:53:58.774461 28266 hybrid_clock.cc:648] HybridClock initialized: now 1790639638774447 us; error 28 us; skew 500 ppm
Sep 28 23:53:58 dist-test-slave-p5bw krb5kdc[28253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:53:58.777066 28266 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:53:58.777809 28266 webserver.cc:460] Webserver started at http://127.0.0.1:41601/ using document root <none> and password file <none>
I20260928 23:53:58.778283 28266 fs_manager.cc:359] Metadata directory not provided
I20260928 23:53:58.778342 28266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:53:58.778502 28266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:53:58.780335 28266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "8c07aa49703e42c3828a12a98b36b8bf"
format_stamp: "Formatted at 2026-09-28 23:53:58 on dist-test-slave-p5bw"
server_key: "fffd5ae034c7bc0a4d707d7fcda49b26"
server_key_iv: "27ec5300f29cd46d5d84552b9a0c2b13"
server_key_version: "encryptionkey@0"
I20260928 23:53:58.780872 28266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "8c07aa49703e42c3828a12a98b36b8bf"
format_stamp: "Formatted at 2026-09-28 23:53:58 on dist-test-slave-p5bw"
server_key: "fffd5ae034c7bc0a4d707d7fcda49b26"
server_key_iv: "27ec5300f29cd46d5d84552b9a0c2b13"
server_key_version: "encryptionkey@0"
I20260928 23:53:58.784264 28266 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 23:53:58.786530 28281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:53:58.787719 28266 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 23:53:58.787851 28266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "8c07aa49703e42c3828a12a98b36b8bf"
format_stamp: "Formatted at 2026-09-28 23:53:58 on dist-test-slave-p5bw"
server_key: "fffd5ae034c7bc0a4d707d7fcda49b26"
server_key_iv: "27ec5300f29cd46d5d84552b9a0c2b13"
server_key_version: "encryptionkey@0"
I20260928 23:53:58.787954 28266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:53:58.792210 28266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:53:58.794826 28266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:53:58.795010 28266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:53:58.801329 28266 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38665
I20260928 23:53:58.801342 28333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38665 every 8 connection(s)
I20260928 23:53:58.802152 28266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:53:58.804636 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:53:58.809644 28334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c07aa49703e42c3828a12a98b36b8bf: Bootstrap starting.
I20260928 23:53:58.811818 28334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c07aa49703e42c3828a12a98b36b8bf: Neither blocks nor log segments found. Creating new log.
I20260928 23:53:58.812001 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 28266
I20260928 23:53:58.812146 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:53:58.812422 6752 external_mini_cluster.cc:1428] Setting key d5d770ca1eed9620675a5755e78eb10c
I20260928 23:53:58.812687 28334 log.cc:824] T 00000000000000000000000000000000 P 8c07aa49703e42c3828a12a98b36b8bf: Log is configured to *not* fsync() on all Append() calls
I20260928 23:53:58.814639 28334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c07aa49703e42c3828a12a98b36b8bf: No bootstrap required, opened a new log
I20260928 23:53:58.817345 28338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:58.813605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.40:53285 (local address 127.0.0.1:38665)
0928 23:53:58.813986 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:58.814000 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:58.814032 (+ 32us) server_negotiation.cc:407] Connection header received
0928 23:53:58.814620 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:58.814635 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:58.814944 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:58.815267 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:58.816006 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:58.816011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:58.816022 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:58.816032 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:58.816080 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:58.816098 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:58.816210 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:58.816306 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 23:53:58.816453 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":99,"threads_started":1}
I20260928 23:53:58.817929 28334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c07aa49703e42c3828a12a98b36b8bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c07aa49703e42c3828a12a98b36b8bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38665 } }
I20260928 23:53:58.818185 28334 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c07aa49703e42c3828a12a98b36b8bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:53:58.818244 28334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c07aa49703e42c3828a12a98b36b8bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c07aa49703e42c3828a12a98b36b8bf, State: Initialized, Role: FOLLOWER
I20260928 23:53:58.818650 28334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c07aa49703e42c3828a12a98b36b8bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c07aa49703e42c3828a12a98b36b8bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38665 } }
I20260928 23:53:58.818763 28334 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c07aa49703e42c3828a12a98b36b8bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:53:58.818845 28334 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c07aa49703e42c3828a12a98b36b8bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:53:58.818970 28334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c07aa49703e42c3828a12a98b36b8bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:53:58.819887 28334 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c07aa49703e42c3828a12a98b36b8bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c07aa49703e42c3828a12a98b36b8bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38665 } }
I20260928 23:53:58.820031 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:53:58.820236 28334 leader_election.cc:302] T 00000000000000000000000000000000 P 8c07aa49703e42c3828a12a98b36b8bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c07aa49703e42c3828a12a98b36b8bf; no voters:
I20260928 23:53:58.820523 28334 leader_election.cc:288] T 00000000000000000000000000000000 P 8c07aa49703e42c3828a12a98b36b8bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:53:58.820633 28339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c07aa49703e42c3828a12a98b36b8bf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:53:58.820859 28339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c07aa49703e42c3828a12a98b36b8bf [term 1 LEADER]: Becoming Leader. State: Replica: 8c07aa49703e42c3828a12a98b36b8bf, State: Running, Role: LEADER
I20260928 23:53:58.821214 28339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c07aa49703e42c3828a12a98b36b8bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c07aa49703e42c3828a12a98b36b8bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38665 } }
I20260928 23:53:58.821729 28334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c07aa49703e42c3828a12a98b36b8bf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:53:58.822764 28340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c07aa49703e42c3828a12a98b36b8bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c07aa49703e42c3828a12a98b36b8bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c07aa49703e42c3828a12a98b36b8bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38665 } } }
I20260928 23:53:58.822919 28340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c07aa49703e42c3828a12a98b36b8bf [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:58.823163 28340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c07aa49703e42c3828a12a98b36b8bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c07aa49703e42c3828a12a98b36b8bf. Latest consensus state: current_term: 1 leader_uuid: "8c07aa49703e42c3828a12a98b36b8bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c07aa49703e42c3828a12a98b36b8bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38665 } } }
I20260928 23:53:58.823261 28340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c07aa49703e42c3828a12a98b36b8bf [sys.catalog]: This master's current role is: LEADER
I20260928 23:53:58.823299 28353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:53:58.824288 28338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:53:58.822252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.40:52129 (local address 127.0.0.1:38665)
0928 23:53:58.822361 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 23:53:58.822367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:53:58.822662 (+ 295us) server_negotiation.cc:407] Connection header received
0928 23:53:58.822851 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:53:58.822856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:53:58.822895 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:53:58.822976 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:53:58.823833 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:53:58.823836 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:53:58.823838 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:53:58.823848 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:53:58.823886 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:53:58.823896 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:53:58.823987 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 23:53:58.824109 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 23:53:58.824153 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 23:53:58.826490 28353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:53:58.831923 28353 catalog_manager.cc:1347] Generated new cluster ID: 25a55851e08044dd8a6e39ca8384a711
I20260928 23:53:58.832011 28353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:53:58.841632 28353 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:53:58.841925 28353 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:53:58.848215 28353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c07aa49703e42c3828a12a98b36b8bf: Generated new TSK 0
I20260928 23:53:58.848882 28353 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:53:58.856489 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 28266
2026-09-28T23:53:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:53:58 dist-test-slave-p5bw krb5kdc[28365](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:53:58 dist-test-slave-p5bw krb5kdc[28365](info): set up 2 sockets
Sep 28 23:53:58 dist-test-slave-p5bw krb5kdc[28365](info): commencing operation
krb5kdc: starting...
W20260928 23:53:59.901798 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:53:59 dist-test-slave-p5bw krb5kdc[28365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:53:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:53:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:54:00.019896 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:44793
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:44793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:00.129298 28378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:00.129609 28378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:00.129691 28378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:00.133735 28378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:54:00.133816 28378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:00.133855 28378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:00.133913 28378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:54:00.133960 28378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:54:00.139235 28378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:44793
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:44793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:00.140280 28378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:00.141357 28378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:54:00.149326 28378 server_base.cc:1034] running on GCE node
W20260928 23:54:00.149291 28383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:00.149299 28384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:00.149292 28386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:00.150126 28378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:00.151125 28378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:00.152305 28378 hybrid_clock.cc:648] HybridClock initialized: now 1790639640152278 us; error 41 us; skew 500 ppm
Sep 28 23:54:00 dist-test-slave-p5bw krb5kdc[28365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639640, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:00.160573 28378 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:54:00.161598 28378 webserver.cc:460] Webserver started at http://127.6.152.62:32771/ using document root <none> and password file <none>
I20260928 23:54:00.162180 28378 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:00.162269 28378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:00.162493 28378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:00.164489 28378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "74fb0b1a1eda47608af05d21ab6b3493"
format_stamp: "Formatted at 2026-09-28 23:54:00 on dist-test-slave-p5bw"
server_key: "6aa52a81fa2c1f67ce5529847f4c0285"
server_key_iv: "09c1857b60aa5f0e4c7fbc4572a3f801"
server_key_version: "encryptionkey@0"
I20260928 23:54:00.165081 28378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "74fb0b1a1eda47608af05d21ab6b3493"
format_stamp: "Formatted at 2026-09-28 23:54:00 on dist-test-slave-p5bw"
server_key: "6aa52a81fa2c1f67ce5529847f4c0285"
server_key_iv: "09c1857b60aa5f0e4c7fbc4572a3f801"
server_key_version: "encryptionkey@0"
I20260928 23:54:00.168720 28378 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 23:54:00.171114 28393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:00.172327 28378 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:54:00.172470 28378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "74fb0b1a1eda47608af05d21ab6b3493"
format_stamp: "Formatted at 2026-09-28 23:54:00 on dist-test-slave-p5bw"
server_key: "6aa52a81fa2c1f67ce5529847f4c0285"
server_key_iv: "09c1857b60aa5f0e4c7fbc4572a3f801"
server_key_version: "encryptionkey@0"
I20260928 23:54:00.172580 28378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:00.197294 28378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:00.200021 28378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:00.200241 28378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:00.207451 28378 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:44793
I20260928 23:54:00.207479 28445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:44793 every 8 connection(s)
I20260928 23:54:00.208389 28378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:54:00.211117 28446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:00.215847 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 28378
I20260928 23:54:00.216007 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:54:00.216320 6752 external_mini_cluster.cc:1428] Setting key 408f00abd006354de47f03ae556628af
I20260928 23:54:00.216809 28446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74fb0b1a1eda47608af05d21ab6b3493: Bootstrap starting.
I20260928 23:54:00.219137 28446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74fb0b1a1eda47608af05d21ab6b3493: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:00.220038 28446 log.cc:824] T 00000000000000000000000000000000 P 74fb0b1a1eda47608af05d21ab6b3493: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:00.222137 28446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74fb0b1a1eda47608af05d21ab6b3493: No bootstrap required, opened a new log
Sep 28 23:54:00 dist-test-slave-p5bw krb5kdc[28365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:54:00.226503 28446 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74fb0b1a1eda47608af05d21ab6b3493 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74fb0b1a1eda47608af05d21ab6b3493" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 44793 } }
I20260928 23:54:00.226739 28446 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74fb0b1a1eda47608af05d21ab6b3493 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:00.226820 28446 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74fb0b1a1eda47608af05d21ab6b3493 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fb0b1a1eda47608af05d21ab6b3493, State: Initialized, Role: FOLLOWER
I20260928 23:54:00.227253 28446 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74fb0b1a1eda47608af05d21ab6b3493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74fb0b1a1eda47608af05d21ab6b3493" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 44793 } }
I20260928 23:54:00.227401 28446 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74fb0b1a1eda47608af05d21ab6b3493 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:54:00.227470 28446 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74fb0b1a1eda47608af05d21ab6b3493 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:54:00.227573 28446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74fb0b1a1eda47608af05d21ab6b3493 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:00.228418 28446 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74fb0b1a1eda47608af05d21ab6b3493 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74fb0b1a1eda47608af05d21ab6b3493" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 44793 } }
I20260928 23:54:00.228751 28446 leader_election.cc:302] T 00000000000000000000000000000000 P 74fb0b1a1eda47608af05d21ab6b3493 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74fb0b1a1eda47608af05d21ab6b3493; no voters:
I20260928 23:54:00.229101 28446 leader_election.cc:288] T 00000000000000000000000000000000 P 74fb0b1a1eda47608af05d21ab6b3493 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:54:00.229206 28451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74fb0b1a1eda47608af05d21ab6b3493 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:00.229434 28451 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74fb0b1a1eda47608af05d21ab6b3493 [term 1 LEADER]: Becoming Leader. State: Replica: 74fb0b1a1eda47608af05d21ab6b3493, State: Running, Role: LEADER
I20260928 23:54:00.229753 28451 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74fb0b1a1eda47608af05d21ab6b3493 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74fb0b1a1eda47608af05d21ab6b3493" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 44793 } }
I20260928 23:54:00.230458 28446 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74fb0b1a1eda47608af05d21ab6b3493 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:54:00.230883 28452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74fb0b1a1eda47608af05d21ab6b3493 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74fb0b1a1eda47608af05d21ab6b3493" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74fb0b1a1eda47608af05d21ab6b3493" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 44793 } } }
I20260928 23:54:00.230908 28453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74fb0b1a1eda47608af05d21ab6b3493 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74fb0b1a1eda47608af05d21ab6b3493. Latest consensus state: current_term: 1 leader_uuid: "74fb0b1a1eda47608af05d21ab6b3493" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74fb0b1a1eda47608af05d21ab6b3493" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 44793 } } }
I20260928 23:54:00.231127 28453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74fb0b1a1eda47608af05d21ab6b3493 [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:00.231756 28449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:00.217816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50811 (local address 127.6.152.62:44793)
0928 23:54:00.218213 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:00.218232 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:00.218275 (+ 43us) server_negotiation.cc:407] Connection header received
0928 23:54:00.218891 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:00.218907 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:00.219295 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:00.219591 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:00.220525 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:00.221664 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:00.222806 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:00.223297 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:00.223414 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:00.226264 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:00.226269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:00.226282 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:00.226308 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:00.228257 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:00.228841 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:00.228848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:00.228853 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:00.229056 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:00.229456 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:00.229461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:00.229464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:00.229942 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:00.230155 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:00.230512 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 23:54:00.230741 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":133,"threads_started":1}
I20260928 23:54:00.231034 28452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74fb0b1a1eda47608af05d21ab6b3493 [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:00.235185 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 23:54:00.235664 28466 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 74fb0b1a1eda47608af05d21ab6b3493: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:54:00.235759 28466 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:54:00.235826 28457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:54:00.236698 28457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:54:00.242106 28457 catalog_manager.cc:1347] Generated new cluster ID: 36bbff5375e44b5493d0c56e39f00f3f
I20260928 23:54:00.242170 28457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:54:00.247030 28449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:00.237426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55329 (local address 127.6.152.62:44793)
0928 23:54:00.237526 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:00.237530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:00.238559 (+ 1029us) server_negotiation.cc:407] Connection header received
0928 23:54:00.238696 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:00.238699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:00.238740 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:00.238832 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:00.239452 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:00.240385 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:00.241608 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:00.241938 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:00.242045 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:00.243663 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:00.243667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:00.243670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:00.243693 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:00.245289 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:00.245811 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:00.245814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:00.245815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:00.245891 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:00.246232 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:00.246235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:00.246237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:00.246404 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:00.246530 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:00.246806 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 23:54:00.246888 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 23:54:00.284336 28457 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:54:00.285331 28457 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:54:00.289251 28457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74fb0b1a1eda47608af05d21ab6b3493: Generated new TSK 0
I20260928 23:54:00.289930 28457 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:54:00.294226 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 28378
W20260928 23:54:00.299628 28363 connection.cc:535] client connection to 127.6.152.62:44793 recv error: Network error: failed to read from TLS socket (remote: 127.6.152.62:44793): Connection reset by peer (error 104)
2026-09-28T23:54:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1438 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:54:00 dist-test-slave-p5bw krb5kdc[28477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:54:00 dist-test-slave-p5bw krb5kdc[28477](info): set up 2 sockets
Sep 28 23:54:00 dist-test-slave-p5bw krb5kdc[28477](info): commencing operation
krb5kdc: starting...
W20260928 23:54:01.317759 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:54:01 dist-test-slave-p5bw krb5kdc[28477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:54:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:54:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:54:01.437659 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:45533
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:42109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:45533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:01.547194 28490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:01.547443 28490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:01.547490 28490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:01.551431 28490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:54:01.551489 28490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:01.551508 28490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:01.551524 28490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:54:01.551539 28490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:54:01.556686 28490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:42109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:45533
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:45533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:01.557708 28490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:01.558689 28490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:01.566736 28496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:01.566776 28495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:01.566764 28490 server_base.cc:1034] running on GCE node
W20260928 23:54:01.566897 28498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:01.567512 28490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:01.568526 28490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:01.569710 28490 hybrid_clock.cc:648] HybridClock initialized: now 1790639641569691 us; error 30 us; skew 500 ppm
Sep 28 23:54:01 dist-test-slave-p5bw krb5kdc[28477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639641, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:01.572238 28490 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:54:01.573118 28490 webserver.cc:460] Webserver started at http://127.6.152.62:34427/ using document root <none> and password file <none>
I20260928 23:54:01.573644 28490 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:01.573731 28490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:01.573956 28490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:01.576164 28490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "cdee48cf592b4d01b9aabec19ee9d751"
format_stamp: "Formatted at 2026-09-28 23:54:01 on dist-test-slave-p5bw"
server_key: "b79d85af9fbaa8df2bc9792809d4406e"
server_key_iv: "76c3c135c193314b46566196c791de3e"
server_key_version: "encryptionkey@0"
I20260928 23:54:01.576716 28490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "cdee48cf592b4d01b9aabec19ee9d751"
format_stamp: "Formatted at 2026-09-28 23:54:01 on dist-test-slave-p5bw"
server_key: "b79d85af9fbaa8df2bc9792809d4406e"
server_key_iv: "76c3c135c193314b46566196c791de3e"
server_key_version: "encryptionkey@0"
I20260928 23:54:01.580235 28490 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:54:01.582520 28505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:01.583762 28490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 23:54:01.583900 28490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "cdee48cf592b4d01b9aabec19ee9d751"
format_stamp: "Formatted at 2026-09-28 23:54:01 on dist-test-slave-p5bw"
server_key: "b79d85af9fbaa8df2bc9792809d4406e"
server_key_iv: "76c3c135c193314b46566196c791de3e"
server_key_version: "encryptionkey@0"
I20260928 23:54:01.584021 28490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:01.611158 28490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:01.614112 28490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:01.614315 28490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:01.621300 28490 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:45533
I20260928 23:54:01.621310 28557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:45533 every 8 connection(s)
I20260928 23:54:01.622231 28490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:54:01.623870 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 28490
I20260928 23:54:01.624045 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:54:01.624396 6752 external_mini_cluster.cc:1428] Setting key 9db7af85b59082f501e3530223fe6a44
I20260928 23:54:01.625167 28558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:01.631524 28558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdee48cf592b4d01b9aabec19ee9d751: Bootstrap starting.
I20260928 23:54:01.633822 28561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:01.625825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33453 (local address 127.6.152.62:45533)
0928 23:54:01.626242 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:01.626263 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:01.626306 (+ 43us) server_negotiation.cc:407] Connection header received
0928 23:54:01.627003 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:01.627020 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:01.627372 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:01.627661 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:01.627998 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:01.629160 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:01.630399 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:01.630922 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:01.631046 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:01.632092 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:01.632097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:01.632109 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:54:01.632498 (+ 389us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:01.632554 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:54:01.632577 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:01.632681 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:01.632980 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 23:54:01.633125 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":139,"threads_started":1}
I20260928 23:54:01.633864 28558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdee48cf592b4d01b9aabec19ee9d751: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:01.634596 28558 log.cc:824] T 00000000000000000000000000000000 P cdee48cf592b4d01b9aabec19ee9d751: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:01.636497 28558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdee48cf592b4d01b9aabec19ee9d751: No bootstrap required, opened a new log
I20260928 23:54:01.636837 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:54:01.641240 28558 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdee48cf592b4d01b9aabec19ee9d751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdee48cf592b4d01b9aabec19ee9d751" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45533 } }
I20260928 23:54:01.641495 28558 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdee48cf592b4d01b9aabec19ee9d751 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:01.641557 28558 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdee48cf592b4d01b9aabec19ee9d751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdee48cf592b4d01b9aabec19ee9d751, State: Initialized, Role: FOLLOWER
I20260928 23:54:01.642007 28558 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdee48cf592b4d01b9aabec19ee9d751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdee48cf592b4d01b9aabec19ee9d751" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45533 } }
I20260928 23:54:01.642158 28558 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdee48cf592b4d01b9aabec19ee9d751 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:54:01.642242 28558 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdee48cf592b4d01b9aabec19ee9d751 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:54:01.642336 28558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdee48cf592b4d01b9aabec19ee9d751 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:01.643477 28561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:01.639005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39467 (local address 127.6.152.62:45533)
0928 23:54:01.639146 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:01.639152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:01.639291 (+ 139us) server_negotiation.cc:407] Connection header received
0928 23:54:01.639390 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:01.639393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:01.639444 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:01.639526 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:01.639757 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:01.640518 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:01.641500 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:01.641824 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:01.641932 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:01.642928 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:01.642932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:01.642935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:54:01.642947 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:01.642977 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:54:01.642989 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:01.643079 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:01.643277 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 23:54:01.643313 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:54:01.643553 28558 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdee48cf592b4d01b9aabec19ee9d751 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdee48cf592b4d01b9aabec19ee9d751" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45533 } }
I20260928 23:54:01.643873 28558 leader_election.cc:302] T 00000000000000000000000000000000 P cdee48cf592b4d01b9aabec19ee9d751 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdee48cf592b4d01b9aabec19ee9d751; no voters:
I20260928 23:54:01.644209 28558 leader_election.cc:288] T 00000000000000000000000000000000 P cdee48cf592b4d01b9aabec19ee9d751 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:54:01.644351 28568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdee48cf592b4d01b9aabec19ee9d751 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:01.644562 28568 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdee48cf592b4d01b9aabec19ee9d751 [term 1 LEADER]: Becoming Leader. State: Replica: cdee48cf592b4d01b9aabec19ee9d751, State: Running, Role: LEADER
I20260928 23:54:01.644872 28568 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdee48cf592b4d01b9aabec19ee9d751 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdee48cf592b4d01b9aabec19ee9d751" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45533 } }
I20260928 23:54:01.645443 28558 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdee48cf592b4d01b9aabec19ee9d751 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:54:01.646224 28569 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdee48cf592b4d01b9aabec19ee9d751 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdee48cf592b4d01b9aabec19ee9d751" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdee48cf592b4d01b9aabec19ee9d751" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45533 } } }
I20260928 23:54:01.646353 28569 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdee48cf592b4d01b9aabec19ee9d751 [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:01.646706 28570 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdee48cf592b4d01b9aabec19ee9d751 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdee48cf592b4d01b9aabec19ee9d751. Latest consensus state: current_term: 1 leader_uuid: "cdee48cf592b4d01b9aabec19ee9d751" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdee48cf592b4d01b9aabec19ee9d751" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45533 } } }
I20260928 23:54:01.646822 28570 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdee48cf592b4d01b9aabec19ee9d751 [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:01.647593 28577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:54:01.650312 28577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:54:01.655583 28577 catalog_manager.cc:1347] Generated new cluster ID: 8aaf907f4fdb4f06b92c4553ba427d75
I20260928 23:54:01.655685 28577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:54:01.662225 28577 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:54:01.663344 28577 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:54:01.669147 28577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdee48cf592b4d01b9aabec19ee9d751: Generated new TSK 0
I20260928 23:54:01.669831 28577 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:54:01.679436 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 28490
W20260928 23:54:01.684197 28475 connection.cc:535] client connection to 127.6.152.62:45533 recv error: Network error: failed to read from TLS socket (remote: 127.6.152.62:45533): Connection reset by peer (error 104)
2026-09-28T23:54:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:54:01 dist-test-slave-p5bw krb5kdc[28589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:54:01 dist-test-slave-p5bw krb5kdc[28589](info): set up 2 sockets
Sep 28 23:54:01 dist-test-slave-p5bw krb5kdc[28589](info): commencing operation
krb5kdc: starting...
W20260928 23:54:02.717723 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:54:02 dist-test-slave-p5bw krb5kdc[28589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:54:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:54:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:54:02.840307 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:36649
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:36649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:02.947589 28602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:02.947868 28602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:02.947943 28602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:02.951939 28602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:54:02.952028 28602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:02.952065 28602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:02.952128 28602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:54:02.952167 28602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:54:02.957335 28602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:36649
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:36649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:02.958349 28602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:02.959319 28602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:02.967389 28607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:02.967409 28608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:02.967389 28610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:02.967734 28602 server_base.cc:1034] running on GCE node
I20260928 23:54:02.968302 28602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:02.969399 28602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:02.970557 28602 hybrid_clock.cc:648] HybridClock initialized: now 1790639642970525 us; error 43 us; skew 500 ppm
Sep 28 23:54:02 dist-test-slave-p5bw krb5kdc[28589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639642, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:02.973309 28602 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:54:02.974202 28602 webserver.cc:460] Webserver started at http://127.6.152.62:41931/ using document root <none> and password file <none>
I20260928 23:54:02.974761 28602 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:02.974853 28602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:02.975070 28602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:02.977083 28602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "227b55a81fe848a19b85032141d9d578"
format_stamp: "Formatted at 2026-09-28 23:54:02 on dist-test-slave-p5bw"
server_key: "329c3a71e0b1aa485e824e0342f93dac"
server_key_iv: "bfa08c6251b6feaf5a331a601eab407e"
server_key_version: "encryptionkey@0"
I20260928 23:54:02.977641 28602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "227b55a81fe848a19b85032141d9d578"
format_stamp: "Formatted at 2026-09-28 23:54:02 on dist-test-slave-p5bw"
server_key: "329c3a71e0b1aa485e824e0342f93dac"
server_key_iv: "bfa08c6251b6feaf5a331a601eab407e"
server_key_version: "encryptionkey@0"
I20260928 23:54:02.981285 28602 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:54:02.983609 28617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:02.984752 28602 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:54:02.984915 28602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "227b55a81fe848a19b85032141d9d578"
format_stamp: "Formatted at 2026-09-28 23:54:02 on dist-test-slave-p5bw"
server_key: "329c3a71e0b1aa485e824e0342f93dac"
server_key_iv: "bfa08c6251b6feaf5a331a601eab407e"
server_key_version: "encryptionkey@0"
I20260928 23:54:02.985039 28602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:02.989318 28602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:02.991930 28602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:02.992133 28602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:02.998492 28602 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:36649
I20260928 23:54:02.998523 28669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:36649 every 8 connection(s)
I20260928 23:54:02.999358 28602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:54:03.001916 28670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:03.006500 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 28602
I20260928 23:54:03.006654 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:54:03.006934 6752 external_mini_cluster.cc:1428] Setting key 18b6105bca9b806274a8642968d31786
I20260928 23:54:03.007295 28670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 227b55a81fe848a19b85032141d9d578: Bootstrap starting.
I20260928 23:54:03.009768 28670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 227b55a81fe848a19b85032141d9d578: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:03.010545 28670 log.cc:824] T 00000000000000000000000000000000 P 227b55a81fe848a19b85032141d9d578: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:03.012085 28673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:03.008302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47071 (local address 127.6.152.62:36649)
0928 23:54:03.008746 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:03.008761 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:03.008796 (+ 35us) server_negotiation.cc:407] Connection header received
0928 23:54:03.009491 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:03.009508 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:03.009778 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:03.010144 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:03.010963 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:03.010967 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:03.010978 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:54:03.010988 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:03.011040 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:54:03.011060 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:03.011173 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:03.011274 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 23:54:03.011424 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":134,"threads_started":1}
I20260928 23:54:03.012447 28670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 227b55a81fe848a19b85032141d9d578: No bootstrap required, opened a new log
I20260928 23:54:03.015018 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:54:03.016599 28670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 227b55a81fe848a19b85032141d9d578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "227b55a81fe848a19b85032141d9d578" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36649 } }
I20260928 23:54:03.016866 28670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 227b55a81fe848a19b85032141d9d578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:03.016952 28670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 227b55a81fe848a19b85032141d9d578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 227b55a81fe848a19b85032141d9d578, State: Initialized, Role: FOLLOWER
I20260928 23:54:03.017490 28670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 227b55a81fe848a19b85032141d9d578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "227b55a81fe848a19b85032141d9d578" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36649 } }
I20260928 23:54:03.017684 28670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 227b55a81fe848a19b85032141d9d578 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:54:03.017786 28670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 227b55a81fe848a19b85032141d9d578 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:54:03.017911 28670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 227b55a81fe848a19b85032141d9d578 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:03.018797 28670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 227b55a81fe848a19b85032141d9d578 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "227b55a81fe848a19b85032141d9d578" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36649 } }
I20260928 23:54:03.018980 28673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:03.017064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56771 (local address 127.6.152.62:36649)
0928 23:54:03.017198 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:03.017202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:03.017467 (+ 265us) server_negotiation.cc:407] Connection header received
0928 23:54:03.017589 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:03.017591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:03.017642 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:03.017734 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:03.018504 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:03.018508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:03.018510 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:54:03.018521 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:03.018553 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:54:03.018563 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:03.018651 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:03.018736 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 23:54:03.018774 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:54:03.019208 28670 leader_election.cc:302] T 00000000000000000000000000000000 P 227b55a81fe848a19b85032141d9d578 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 227b55a81fe848a19b85032141d9d578; no voters:
I20260928 23:54:03.019563 28670 leader_election.cc:288] T 00000000000000000000000000000000 P 227b55a81fe848a19b85032141d9d578 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:54:03.019641 28680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 227b55a81fe848a19b85032141d9d578 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:03.019843 28680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 227b55a81fe848a19b85032141d9d578 [term 1 LEADER]: Becoming Leader. State: Replica: 227b55a81fe848a19b85032141d9d578, State: Running, Role: LEADER
I20260928 23:54:03.020172 28680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 227b55a81fe848a19b85032141d9d578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "227b55a81fe848a19b85032141d9d578" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36649 } }
I20260928 23:54:03.020771 28670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 227b55a81fe848a19b85032141d9d578 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:54:03.022017 28682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 227b55a81fe848a19b85032141d9d578 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 227b55a81fe848a19b85032141d9d578. Latest consensus state: current_term: 1 leader_uuid: "227b55a81fe848a19b85032141d9d578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "227b55a81fe848a19b85032141d9d578" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36649 } } }
I20260928 23:54:03.022056 28681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 227b55a81fe848a19b85032141d9d578 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "227b55a81fe848a19b85032141d9d578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "227b55a81fe848a19b85032141d9d578" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 36649 } } }
I20260928 23:54:03.022148 28682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 227b55a81fe848a19b85032141d9d578 [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:03.022149 28681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 227b55a81fe848a19b85032141d9d578 [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:03.023016 28689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:54:03.025967 28689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:54:03.030834 28689 catalog_manager.cc:1347] Generated new cluster ID: 59ea4f98e00f4bc9a7c539c5bf8f0feb
I20260928 23:54:03.030905 28689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:54:03.057416 28689 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:54:03.057780 28689 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:54:03.066094 28689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 227b55a81fe848a19b85032141d9d578: Generated new TSK 0
I20260928 23:54:03.066803 28689 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:54:03.079519 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 28602
2026-09-28T23:54:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:54:03 dist-test-slave-p5bw krb5kdc[28701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:54:03 dist-test-slave-p5bw krb5kdc[28701](info): set up 2 sockets
Sep 28 23:54:03 dist-test-slave-p5bw krb5kdc[28701](info): commencing operation
krb5kdc: starting...
W20260928 23:54:04.145666 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:54:04 dist-test-slave-p5bw krb5kdc[28701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:54:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:54:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:54:04.263507 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:41235
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:41235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:04.373548 28714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:04.373778 28714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:04.373818 28714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:04.377631 28714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:54:04.377681 28714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:04.377700 28714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:04.377717 28714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:54:04.377731 28714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:54:04.382684 28714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:41235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:41235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:04.383598 28714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:04.384498 28714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:04.392632 28720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:04.392659 28719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:04.392891 28714 server_base.cc:1034] running on GCE node
W20260928 23:54:04.392659 28722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:04.393512 28714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:04.394544 28714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:04.395730 28714 hybrid_clock.cc:648] HybridClock initialized: now 1790639644395714 us; error 34 us; skew 500 ppm
Sep 28 23:54:04 dist-test-slave-p5bw krb5kdc[28701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639644, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:04.398411 28714 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:54:04.399240 28714 webserver.cc:460] Webserver started at http://127.6.152.62:32953/ using document root <none> and password file <none>
I20260928 23:54:04.399771 28714 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:04.399861 28714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:04.400079 28714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:04.402073 28714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "804389fa56be48449f4d92df1a6cd1f6"
format_stamp: "Formatted at 2026-09-28 23:54:04 on dist-test-slave-p5bw"
server_key: "19febae826edd279b94bd8905e804596"
server_key_iv: "2a30b1a9753d0d6e2946cd4e14710b4a"
server_key_version: "encryptionkey@0"
I20260928 23:54:04.402623 28714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "804389fa56be48449f4d92df1a6cd1f6"
format_stamp: "Formatted at 2026-09-28 23:54:04 on dist-test-slave-p5bw"
server_key: "19febae826edd279b94bd8905e804596"
server_key_iv: "2a30b1a9753d0d6e2946cd4e14710b4a"
server_key_version: "encryptionkey@0"
I20260928 23:54:04.406124 28714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:54:04.408357 28729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:04.409619 28714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:54:04.409754 28714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "804389fa56be48449f4d92df1a6cd1f6"
format_stamp: "Formatted at 2026-09-28 23:54:04 on dist-test-slave-p5bw"
server_key: "19febae826edd279b94bd8905e804596"
server_key_iv: "2a30b1a9753d0d6e2946cd4e14710b4a"
server_key_version: "encryptionkey@0"
I20260928 23:54:04.409862 28714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:04.431751 28714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:04.435137 28714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:04.435348 28714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:04.442301 28714 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:41235
I20260928 23:54:04.442324 28781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:41235 every 8 connection(s)
I20260928 23:54:04.443204 28714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:54:04.445842 28782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:04.449492 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 28714
I20260928 23:54:04.449633 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:54:04.449900 6752 external_mini_cluster.cc:1428] Setting key 33d490c20cc7f8539361f2ba74aa6fbc
I20260928 23:54:04.451519 28782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 804389fa56be48449f4d92df1a6cd1f6: Bootstrap starting.
I20260928 23:54:04.453693 28782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 804389fa56be48449f4d92df1a6cd1f6: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:04.454542 28782 log.cc:824] T 00000000000000000000000000000000 P 804389fa56be48449f4d92df1a6cd1f6: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:04.456419 28782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 804389fa56be48449f4d92df1a6cd1f6: No bootstrap required, opened a new log
Sep 28 23:54:04 dist-test-slave-p5bw krb5kdc[28701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:54:04.460691 28782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 804389fa56be48449f4d92df1a6cd1f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804389fa56be48449f4d92df1a6cd1f6" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41235 } }
I20260928 23:54:04.460978 28782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 804389fa56be48449f4d92df1a6cd1f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:04.461056 28782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 804389fa56be48449f4d92df1a6cd1f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804389fa56be48449f4d92df1a6cd1f6, State: Initialized, Role: FOLLOWER
I20260928 23:54:04.461488 28782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 804389fa56be48449f4d92df1a6cd1f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804389fa56be48449f4d92df1a6cd1f6" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41235 } }
I20260928 23:54:04.461604 28782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 804389fa56be48449f4d92df1a6cd1f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:54:04.461690 28782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 804389fa56be48449f4d92df1a6cd1f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:54:04.461808 28782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 804389fa56be48449f4d92df1a6cd1f6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:04.462598 28782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 804389fa56be48449f4d92df1a6cd1f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804389fa56be48449f4d92df1a6cd1f6" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41235 } }
I20260928 23:54:04.462925 28782 leader_election.cc:302] T 00000000000000000000000000000000 P 804389fa56be48449f4d92df1a6cd1f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 804389fa56be48449f4d92df1a6cd1f6; no voters:
I20260928 23:54:04.463258 28782 leader_election.cc:288] T 00000000000000000000000000000000 P 804389fa56be48449f4d92df1a6cd1f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:54:04.463307 28787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 804389fa56be48449f4d92df1a6cd1f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:04.463553 28787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 804389fa56be48449f4d92df1a6cd1f6 [term 1 LEADER]: Becoming Leader. State: Replica: 804389fa56be48449f4d92df1a6cd1f6, State: Running, Role: LEADER
I20260928 23:54:04.463871 28787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 804389fa56be48449f4d92df1a6cd1f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804389fa56be48449f4d92df1a6cd1f6" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41235 } }
I20260928 23:54:04.464512 28782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 804389fa56be48449f4d92df1a6cd1f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:54:04.464560 28785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:04.451257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48609 (local address 127.6.152.62:41235)
0928 23:54:04.451704 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:04.451720 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:04.451755 (+ 35us) server_negotiation.cc:407] Connection header received
0928 23:54:04.452359 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:04.452374 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:04.452690 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:04.453061 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:04.453881 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:04.455271 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:04.456147 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:04.456838 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:04.457003 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:04.459708 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:04.459712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:04.459724 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:04.459749 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:04.461611 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:04.462119 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:04.462125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:04.462131 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:04.462244 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:04.462535 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:04.462538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:04.462540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:04.462975 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:04.463183 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:04.463491 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 23:54:04.463795 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":127,"threads_started":1}
I20260928 23:54:04.465920 28789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 804389fa56be48449f4d92df1a6cd1f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "804389fa56be48449f4d92df1a6cd1f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804389fa56be48449f4d92df1a6cd1f6" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41235 } } }
I20260928 23:54:04.466050 28789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 804389fa56be48449f4d92df1a6cd1f6 [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:04.467196 28799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:54:04.467379 28788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 804389fa56be48449f4d92df1a6cd1f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 804389fa56be48449f4d92df1a6cd1f6. Latest consensus state: current_term: 1 leader_uuid: "804389fa56be48449f4d92df1a6cd1f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804389fa56be48449f4d92df1a6cd1f6" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41235 } } }
I20260928 23:54:04.467518 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:54:04.467545 28788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 804389fa56be48449f4d92df1a6cd1f6 [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:04.470294 28799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:54:04.475942 28799 catalog_manager.cc:1347] Generated new cluster ID: fd4a8a0ef98e43089dfff99fe852a5dd
I20260928 23:54:04.476310 28799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:54:04.478906 28785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:04.469574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60363 (local address 127.6.152.62:41235)
0928 23:54:04.469697 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:04.469701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:04.469983 (+ 282us) server_negotiation.cc:407] Connection header received
0928 23:54:04.470080 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:04.470083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:04.470126 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:04.470210 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:04.470986 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:04.471721 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:04.472907 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:04.473241 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:04.473345 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:04.475418 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:04.475422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:04.475424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:04.475448 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:04.477253 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:04.477721 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:04.477724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:04.477726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:04.477802 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:04.478087 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:04.478089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:04.478091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:04.478269 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:04.478378 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:04.478633 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 23:54:04.478739 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:54:04.485155 28799 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:54:04.485965 28799 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:54:04.491384 28799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 804389fa56be48449f4d92df1a6cd1f6: Generated new TSK 0
I20260928 23:54:04.491992 28799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:54:04.498543 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 28714
2026-09-28T23:54:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:54:04 dist-test-slave-p5bw krb5kdc[28813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:54:04 dist-test-slave-p5bw krb5kdc[28813](info): set up 2 sockets
Sep 28 23:54:04 dist-test-slave-p5bw krb5kdc[28813](info): commencing operation
krb5kdc: starting...
W20260928 23:54:05.537698 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:54:05 dist-test-slave-p5bw krb5kdc[28813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:54:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:54:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:54:05.654146 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:42121
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:39015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:42121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:05.763664 28826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:05.763924 28826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:05.764009 28826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:05.767953 28826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:54:05.768013 28826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:05.768031 28826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:05.768093 28826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:54:05.768126 28826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:54:05.773340 28826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:39015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:42121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:42121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:05.774333 28826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:05.775319 28826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:05.783468 28831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:05.783445 28832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:05.783494 28826 server_base.cc:1034] running on GCE node
W20260928 23:54:05.783445 28834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:05.784188 28826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:05.785225 28826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:05.786402 28826 hybrid_clock.cc:648] HybridClock initialized: now 1790639645786388 us; error 35 us; skew 500 ppm
Sep 28 23:54:05 dist-test-slave-p5bw krb5kdc[28813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639645, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:05.795252 28826 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:54:05.796051 28826 webserver.cc:460] Webserver started at http://127.6.152.62:37661/ using document root <none> and password file <none>
I20260928 23:54:05.796541 28826 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:05.796598 28826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:05.796754 28826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:05.798748 28826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "807398ca9d3b44eba68eebd4a6cde0bc"
format_stamp: "Formatted at 2026-09-28 23:54:05 on dist-test-slave-p5bw"
server_key: "17636b82afe58c7e5483d519f2ed2352"
server_key_iv: "8ebeb0b87245b39da029513d63c12be5"
server_key_version: "encryptionkey@0"
I20260928 23:54:05.799252 28826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "807398ca9d3b44eba68eebd4a6cde0bc"
format_stamp: "Formatted at 2026-09-28 23:54:05 on dist-test-slave-p5bw"
server_key: "17636b82afe58c7e5483d519f2ed2352"
server_key_iv: "8ebeb0b87245b39da029513d63c12be5"
server_key_version: "encryptionkey@0"
I20260928 23:54:05.802963 28826 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 23:54:05.805264 28841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:05.806463 28826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:54:05.806600 28826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "807398ca9d3b44eba68eebd4a6cde0bc"
format_stamp: "Formatted at 2026-09-28 23:54:05 on dist-test-slave-p5bw"
server_key: "17636b82afe58c7e5483d519f2ed2352"
server_key_iv: "8ebeb0b87245b39da029513d63c12be5"
server_key_version: "encryptionkey@0"
I20260928 23:54:05.806703 28826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:05.825857 28826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:05.828711 28826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:05.828997 28826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:05.836045 28826 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:42121
I20260928 23:54:05.836071 28893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:42121 every 8 connection(s)
I20260928 23:54:05.837003 28826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:54:05.839636 28894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:05.840163 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 28826
I20260928 23:54:05.840298 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:54:05.840587 6752 external_mini_cluster.cc:1428] Setting key 3d4941a885cfa6547ea9ff33d8c70978
I20260928 23:54:05.845476 28894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 807398ca9d3b44eba68eebd4a6cde0bc: Bootstrap starting.
I20260928 23:54:05.847692 28894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 807398ca9d3b44eba68eebd4a6cde0bc: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:05.848383 28894 log.cc:824] T 00000000000000000000000000000000 P 807398ca9d3b44eba68eebd4a6cde0bc: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:05.849315 28897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:05.842005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49123 (local address 127.6.152.62:42121)
0928 23:54:05.842423 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:05.842437 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:05.842471 (+ 34us) server_negotiation.cc:407] Connection header received
0928 23:54:05.843086 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:05.843102 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:05.843459 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:05.843788 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:05.844172 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:05.845215 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:05.846152 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:05.846615 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:05.846798 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:05.847608 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:05.847613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:05.847625 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:54:05.848030 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:05.848081 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:54:05.848103 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:05.848212 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:05.848360 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 23:54:05.848560 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":125,"threads_started":1}
I20260928 23:54:05.850486 28894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 807398ca9d3b44eba68eebd4a6cde0bc: No bootstrap required, opened a new log
I20260928 23:54:05.852046 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:54:05.854270 28894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 807398ca9d3b44eba68eebd4a6cde0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807398ca9d3b44eba68eebd4a6cde0bc" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 42121 } }
I20260928 23:54:05.854591 28894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 807398ca9d3b44eba68eebd4a6cde0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:05.854656 28894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 807398ca9d3b44eba68eebd4a6cde0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 807398ca9d3b44eba68eebd4a6cde0bc, State: Initialized, Role: FOLLOWER
I20260928 23:54:05.855094 28894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 807398ca9d3b44eba68eebd4a6cde0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807398ca9d3b44eba68eebd4a6cde0bc" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 42121 } }
I20260928 23:54:05.855253 28894 raft_consensus.cc:388] T 00000000000000000000000000000000 P 807398ca9d3b44eba68eebd4a6cde0bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:54:05.855341 28894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 807398ca9d3b44eba68eebd4a6cde0bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:54:05.855445 28894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 807398ca9d3b44eba68eebd4a6cde0bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:05.856285 28894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 807398ca9d3b44eba68eebd4a6cde0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807398ca9d3b44eba68eebd4a6cde0bc" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 42121 } }
I20260928 23:54:05.856627 28894 leader_election.cc:302] T 00000000000000000000000000000000 P 807398ca9d3b44eba68eebd4a6cde0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 807398ca9d3b44eba68eebd4a6cde0bc; no voters:
I20260928 23:54:05.857012 28894 leader_election.cc:288] T 00000000000000000000000000000000 P 807398ca9d3b44eba68eebd4a6cde0bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:54:05.857079 28904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 807398ca9d3b44eba68eebd4a6cde0bc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:05.857270 28904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 807398ca9d3b44eba68eebd4a6cde0bc [term 1 LEADER]: Becoming Leader. State: Replica: 807398ca9d3b44eba68eebd4a6cde0bc, State: Running, Role: LEADER
I20260928 23:54:05.857554 28904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 807398ca9d3b44eba68eebd4a6cde0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807398ca9d3b44eba68eebd4a6cde0bc" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 42121 } }
I20260928 23:54:05.858134 28897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:05.854076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42247 (local address 127.6.152.62:42121)
0928 23:54:05.854197 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:05.854203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:05.854373 (+ 170us) server_negotiation.cc:407] Connection header received
0928 23:54:05.854489 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:05.854491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:05.854525 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:05.854617 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:05.854851 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:05.855444 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:05.856241 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:05.856566 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:05.856676 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:05.857593 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:05.857596 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:05.857599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:54:05.857611 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:05.857638 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:54:05.857651 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:05.857694 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:05.857934 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 23:54:05.857983 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:54:05.858261 28894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 807398ca9d3b44eba68eebd4a6cde0bc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:54:05.859228 28905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 807398ca9d3b44eba68eebd4a6cde0bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 807398ca9d3b44eba68eebd4a6cde0bc. Latest consensus state: current_term: 1 leader_uuid: "807398ca9d3b44eba68eebd4a6cde0bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807398ca9d3b44eba68eebd4a6cde0bc" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 42121 } } }
I20260928 23:54:05.859368 28905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 807398ca9d3b44eba68eebd4a6cde0bc [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:05.859676 28906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 807398ca9d3b44eba68eebd4a6cde0bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "807398ca9d3b44eba68eebd4a6cde0bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807398ca9d3b44eba68eebd4a6cde0bc" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 42121 } } }
I20260928 23:54:05.859747 28913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:54:05.859764 28906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 807398ca9d3b44eba68eebd4a6cde0bc [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:05.862763 28913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:54:05.868165 28913 catalog_manager.cc:1347] Generated new cluster ID: d1d65afe4e924533877a40584a34ade5
I20260928 23:54:05.868225 28913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:54:05.890717 28913 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:54:05.891705 28913 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:54:05.904496 28913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 807398ca9d3b44eba68eebd4a6cde0bc: Generated new TSK 0
I20260928 23:54:05.905278 28913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:54:05.916328 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 28826
2026-09-28T23:54:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:54:05 dist-test-slave-p5bw krb5kdc[28925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:54:05 dist-test-slave-p5bw krb5kdc[28925](info): set up 2 sockets
Sep 28 23:54:05 dist-test-slave-p5bw krb5kdc[28925](info): commencing operation
krb5kdc: starting...
W20260928 23:54:06.945834 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:54:07 dist-test-slave-p5bw krb5kdc[28925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:54:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:54:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:54:07.067741 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:40279
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:39333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:40279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:07.175985 28938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:07.176261 28938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:07.176338 28938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:07.180253 28938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:54:07.180341 28938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:07.180382 28938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:07.180439 28938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:54:07.180478 28938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:54:07.185555 28938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:39333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:40279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:40279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:07.186525 28938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:07.187490 28938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:07.195785 28944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:07.195668 28943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:07.195672 28946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:07.195925 28938 server_base.cc:1034] running on GCE node
I20260928 23:54:07.196563 28938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:07.197634 28938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:07.198825 28938 hybrid_clock.cc:648] HybridClock initialized: now 1790639647198798 us; error 36 us; skew 500 ppm
Sep 28 23:54:07 dist-test-slave-p5bw krb5kdc[28925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639647, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:07.201519 28938 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:54:07.202397 28938 webserver.cc:460] Webserver started at http://127.6.152.62:39503/ using document root <none> and password file <none>
I20260928 23:54:07.202942 28938 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:07.203033 28938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:07.203292 28938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:07.205325 28938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "a57fd4ae4943413685e1ec892ada8b11"
format_stamp: "Formatted at 2026-09-28 23:54:07 on dist-test-slave-p5bw"
server_key: "ad61833b797ad1cb78c3c66d5399c8f9"
server_key_iv: "4dd54612faa11aa2ba0cb9de45897a33"
server_key_version: "encryptionkey@0"
I20260928 23:54:07.205868 28938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "a57fd4ae4943413685e1ec892ada8b11"
format_stamp: "Formatted at 2026-09-28 23:54:07 on dist-test-slave-p5bw"
server_key: "ad61833b797ad1cb78c3c66d5399c8f9"
server_key_iv: "4dd54612faa11aa2ba0cb9de45897a33"
server_key_version: "encryptionkey@0"
I20260928 23:54:07.209426 28938 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 23:54:07.211675 28953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:07.212931 28938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:54:07.213060 28938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "a57fd4ae4943413685e1ec892ada8b11"
format_stamp: "Formatted at 2026-09-28 23:54:07 on dist-test-slave-p5bw"
server_key: "ad61833b797ad1cb78c3c66d5399c8f9"
server_key_iv: "4dd54612faa11aa2ba0cb9de45897a33"
server_key_version: "encryptionkey@0"
I20260928 23:54:07.213165 28938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:07.217626 28938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:07.220214 28938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:07.220408 28938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:07.227425 28938 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:40279
I20260928 23:54:07.227452 29005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:40279 every 8 connection(s)
I20260928 23:54:07.228284 28938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:54:07.230885 29006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:07.233325 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 28938
I20260928 23:54:07.233459 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:54:07.233721 6752 external_mini_cluster.cc:1428] Setting key 874ba9115350fbe152e9ec4779b3e2d3
I20260928 23:54:07.236438 29006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a57fd4ae4943413685e1ec892ada8b11: Bootstrap starting.
I20260928 23:54:07.238538 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:07.234938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35765 (local address 127.6.152.62:40279)
0928 23:54:07.235312 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:07.235326 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:07.235365 (+ 39us) server_negotiation.cc:407] Connection header received
0928 23:54:07.236011 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:07.236028 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:07.236291 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:07.236602 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:07.237389 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:07.237394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:07.237405 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:54:07.237414 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:07.237463 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:54:07.237481 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:07.237582 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:07.237712 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 23:54:07.237869 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":128,"threads_started":1}
I20260928 23:54:07.238965 29006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a57fd4ae4943413685e1ec892ada8b11: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:07.239650 29006 log.cc:824] T 00000000000000000000000000000000 P a57fd4ae4943413685e1ec892ada8b11: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:07.241174 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:54:07.241460 29006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a57fd4ae4943413685e1ec892ada8b11: No bootstrap required, opened a new log
I20260928 23:54:07.244735 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:07.243149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38089 (local address 127.6.152.62:40279)
0928 23:54:07.243279 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:07.243288 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:07.243408 (+ 120us) server_negotiation.cc:407] Connection header received
0928 23:54:07.243587 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:07.243590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:07.243618 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:07.243688 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:07.244349 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:07.244352 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:07.244353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:54:07.244363 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:07.244384 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:54:07.244394 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:07.244458 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:07.244579 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 23:54:07.244606 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:54:07.245464 29006 raft_consensus.cc:348] T 00000000000000000000000000000000 P a57fd4ae4943413685e1ec892ada8b11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57fd4ae4943413685e1ec892ada8b11" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40279 } }
I20260928 23:54:07.245692 29006 raft_consensus.cc:374] T 00000000000000000000000000000000 P a57fd4ae4943413685e1ec892ada8b11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:54:07.245748 29006 raft_consensus.cc:729] T 00000000000000000000000000000000 P a57fd4ae4943413685e1ec892ada8b11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57fd4ae4943413685e1ec892ada8b11, State: Initialized, Role: FOLLOWER
I20260928 23:54:07.246127 29006 consensus_queue.cc:260] T 00000000000000000000000000000000 P a57fd4ae4943413685e1ec892ada8b11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57fd4ae4943413685e1ec892ada8b11" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40279 } }
I20260928 23:54:07.246274 29006 raft_consensus.cc:388] T 00000000000000000000000000000000 P a57fd4ae4943413685e1ec892ada8b11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:54:07.246342 29006 raft_consensus.cc:482] T 00000000000000000000000000000000 P a57fd4ae4943413685e1ec892ada8b11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:54:07.246439 29006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a57fd4ae4943413685e1ec892ada8b11 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:07.247274 29006 raft_consensus.cc:504] T 00000000000000000000000000000000 P a57fd4ae4943413685e1ec892ada8b11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57fd4ae4943413685e1ec892ada8b11" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40279 } }
I20260928 23:54:07.247604 29006 leader_election.cc:302] T 00000000000000000000000000000000 P a57fd4ae4943413685e1ec892ada8b11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a57fd4ae4943413685e1ec892ada8b11; no voters:
I20260928 23:54:07.247887 29006 leader_election.cc:288] T 00000000000000000000000000000000 P a57fd4ae4943413685e1ec892ada8b11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:54:07.248019 29016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a57fd4ae4943413685e1ec892ada8b11 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:07.248225 29016 raft_consensus.cc:686] T 00000000000000000000000000000000 P a57fd4ae4943413685e1ec892ada8b11 [term 1 LEADER]: Becoming Leader. State: Replica: a57fd4ae4943413685e1ec892ada8b11, State: Running, Role: LEADER
I20260928 23:54:07.248546 29016 consensus_queue.cc:237] T 00000000000000000000000000000000 P a57fd4ae4943413685e1ec892ada8b11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57fd4ae4943413685e1ec892ada8b11" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40279 } }
I20260928 23:54:07.249083 29006 sys_catalog.cc:564] T 00000000000000000000000000000000 P a57fd4ae4943413685e1ec892ada8b11 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:54:07.250382 29017 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57fd4ae4943413685e1ec892ada8b11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a57fd4ae4943413685e1ec892ada8b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57fd4ae4943413685e1ec892ada8b11" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40279 } } }
I20260928 23:54:07.250517 29017 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57fd4ae4943413685e1ec892ada8b11 [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:07.250382 29018 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57fd4ae4943413685e1ec892ada8b11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a57fd4ae4943413685e1ec892ada8b11. Latest consensus state: current_term: 1 leader_uuid: "a57fd4ae4943413685e1ec892ada8b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57fd4ae4943413685e1ec892ada8b11" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 40279 } } }
I20260928 23:54:07.250864 29018 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57fd4ae4943413685e1ec892ada8b11 [sys.catalog]: This master's current role is: LEADER
W20260928 23:54:07.253741 29031 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a57fd4ae4943413685e1ec892ada8b11: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:54:07.253823 29031 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:54:07.253901 29027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:54:07.254578 29027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:54:07.259663 29027 catalog_manager.cc:1347] Generated new cluster ID: 11857811d078442e86d32ddbaad7b6d2
I20260928 23:54:07.259958 29027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:54:07.297740 29027 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:54:07.298087 29027 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:54:07.302851 29027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a57fd4ae4943413685e1ec892ada8b11: Generated new TSK 0
I20260928 23:54:07.303610 29027 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:54:07.307669 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 28938
2026-09-28T23:54:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1391 ms)
[----------] 18 tests from AuthTokenIssuingTest (25153 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:54:07 dist-test-slave-p5bw krb5kdc[29037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:54:07 dist-test-slave-p5bw krb5kdc[29037](info): set up 2 sockets
Sep 28 23:54:07 dist-test-slave-p5bw krb5kdc[29037](info): commencing operation
krb5kdc: starting...
W20260928 23:54:08.377859 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:54:08 dist-test-slave-p5bw krb5kdc[29037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:54:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:54:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:54:08.500041 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:41363
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:41863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:41363,127.6.152.61:38531,127.6.152.60:42263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:08.608789 29050 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:54:08.609138 29050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:08.609206 29050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:08.609282 29050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:08.613247 29050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:54:08.613338 29050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:08.613376 29050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:08.613413 29050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:54:08.613451 29050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:54:08.618556 29050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:41863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:41363,127.6.152.61:38531,127.6.152.60:42263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:41363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:08.619599 29050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:08.620582 29050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:08.629158 29058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:08.629200 29055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:08.629184 29056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:08.629316 29050 server_base.cc:1034] running on GCE node
I20260928 23:54:08.629914 29050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:08.631018 29050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:08.632203 29050 hybrid_clock.cc:648] HybridClock initialized: now 1790639648632172 us; error 44 us; skew 500 ppm
Sep 28 23:54:08 dist-test-slave-p5bw krb5kdc[29037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639648, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:08.634922 29050 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:54:08.635912 29050 webserver.cc:460] Webserver started at http://127.6.152.62:35241/ using document root <none> and password file <none>
I20260928 23:54:08.636482 29050 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:08.636541 29050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:08.636772 29050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:08.638780 29050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "ae8b408b7f6c42a297d449133d8b73ab"
format_stamp: "Formatted at 2026-09-28 23:54:08 on dist-test-slave-p5bw"
server_key: "56977b36f25efd0c7f6dabc40372b544"
server_key_iv: "858ecb9f8a8552d9ce1a2dfc9ee1cdeb"
server_key_version: "encryptionkey@0"
I20260928 23:54:08.639281 29050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "ae8b408b7f6c42a297d449133d8b73ab"
format_stamp: "Formatted at 2026-09-28 23:54:08 on dist-test-slave-p5bw"
server_key: "56977b36f25efd0c7f6dabc40372b544"
server_key_iv: "858ecb9f8a8552d9ce1a2dfc9ee1cdeb"
server_key_version: "encryptionkey@0"
I20260928 23:54:08.642915 29050 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 23:54:08.645210 29065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:08.646420 29050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:54:08.646513 29050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "ae8b408b7f6c42a297d449133d8b73ab"
format_stamp: "Formatted at 2026-09-28 23:54:08 on dist-test-slave-p5bw"
server_key: "56977b36f25efd0c7f6dabc40372b544"
server_key_iv: "858ecb9f8a8552d9ce1a2dfc9ee1cdeb"
server_key_version: "encryptionkey@0"
I20260928 23:54:08.646592 29050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:08.663820 29050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:08.666996 29050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:08.667138 29050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:08.673504 29050 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:41363
I20260928 23:54:08.673774 29117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:41363 every 8 connection(s)
I20260928 23:54:08.674453 29050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:54:08.675930 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 29050
I20260928 23:54:08.676074 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:54:08.676357 6752 external_mini_cluster.cc:1428] Setting key 7cbd511cd874d726554781ee29589f6e
I20260928 23:54:08.677258 29118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:08.682688 29118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41363 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.152.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.61@KRBTEST.COM" created.
Sep 28 23:54:08 dist-test-slave-p5bw krb5kdc[29037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639648, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:54:08.698859 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:08.685479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:41363 (local address 127.0.0.1:45874)
0928 23:54:08.685959 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:08.686005 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:08.686852 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:08.687569 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:08.687573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:08.688072 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:08.688427 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:08.688442 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:08.690113 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:08.690116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:08.691049 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:08.691063 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:08.691710 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:08.691718 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:08.691886 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:08.692501 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:08.692529 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:08.694170 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:08.696058 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:08.696068 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:08.696077 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:08.696294 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:08.696652 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:08.696654 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:08.696657 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:08.696764 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:08.697418 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:08.697424 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:08.697545 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 23:54:08.697679 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 23:54:08.698023 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":148,"threads_started":1}
I20260928 23:54:08.698870 29122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:08.685479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45874 (local address 127.6.152.62:41363)
0928 23:54:08.685917 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:08.685931 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:08.686119 (+ 188us) server_negotiation.cc:407] Connection header received
0928 23:54:08.687145 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:08.687149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:08.687382 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:08.687541 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:08.688585 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:08.689988 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:08.691179 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:08.691543 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:08.691661 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:08.694319 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:08.694325 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:08.694334 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:08.694353 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:08.695936 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:08.696419 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:08.696424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:08.696426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:08.696529 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:08.696879 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:08.696882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:08.696883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:08.697271 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:08.697407 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:08.697831 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 23:54:08.698107 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":113,"server-negotiator.queue_time_us":297,"spinlock_wait_cycles":37760,"thread_start_us":165,"threads_started":1}
I20260928 23:54:08.701330 29118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.61" port: 38531 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:54:08.701961 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:08.701763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:38531 (local address 127.0.0.1:39530)
0928 23:54:08.701845 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:08.701910 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.61:38531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 23:54:08.702149 29066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.152.61:38531: connect: Connection refused (error 111)
I20260928 23:54:08.702931 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:08.702737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:38531 (local address 127.0.0.1:39544)
0928 23:54:08.702835 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:08.702883 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.61:38531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 23:54:08.703083 29118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.61:38531: Network error: Client connection negotiation failed: client connection to 127.6.152.61:38531: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.61.keytab.
Entry for principal kudu/127.6.152.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.61.keytab.
I20260928 23:54:08.704468 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.61:38531
--webserver_interface=127.6.152.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.61.keytab
--principal=kudu/127.6.152.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:41863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:41363,127.6.152.61:38531,127.6.152.60:42263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:54:08.752292 29118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.61" port: 38531 } attempt: 1
I20260928 23:54:08.753160 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:08.752868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:38531 (local address 127.0.0.1:39556)
0928 23:54:08.753020 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:08.753083 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.61:38531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 23:54:08.753887 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:08.753675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:38531 (local address 127.0.0.1:39568)
0928 23:54:08.753780 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:08.753830 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.61:38531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 23:54:08.754011 29118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.61:38531: Network error: Client connection negotiation failed: client connection to 127.6.152.61:38531: connect: Connection refused (error 111)
W20260928 23:54:08.810657 29125 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:54:08.810986 29125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:08.811060 29125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:08.811142 29125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:08.815153 29125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:54:08.815237 29125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:08.815275 29125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:08.815335 29125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:54:08.815374 29125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:54:08.820439 29125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:41863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:41363,127.6.152.61:38531,127.6.152.60:42263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.61.keytab
--principal=kudu/127.6.152.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.61:38531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.152.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:08.821512 29125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:08.822162 29118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.61" port: 38531 } attempt: 2
I20260928 23:54:08.822548 29125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:54:08.822978 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:08.822708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:38531 (local address 127.0.0.1:39582)
0928 23:54:08.822841 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:08.822906 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.61:38531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 23:54:08.823643 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:08.823433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:38531 (local address 127.0.0.1:39586)
0928 23:54:08.823538 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:08.823583 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.61:38531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 23:54:08.823756 29118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.61:38531: Network error: Client connection negotiation failed: client connection to 127.6.152.61:38531: connect: Connection refused (error 111)
I20260928 23:54:08.830927 29125 server_base.cc:1034] running on GCE node
W20260928 23:54:08.830919 29131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:08.830893 29130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:08.830888 29133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:08.831642 29125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:08.832692 29125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:08.833875 29125 hybrid_clock.cc:648] HybridClock initialized: now 1790639648833845 us; error 47 us; skew 500 ppm
Sep 28 23:54:08 dist-test-slave-p5bw krb5kdc[29037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639648, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:08.836673 29125 init.cc:377] Logged in from keytab as kudu/127.6.152.61@KRBTEST.COM (short username kudu)
I20260928 23:54:08.837522 29125 webserver.cc:460] Webserver started at http://127.6.152.61:38723/ using document root <none> and password file <none>
I20260928 23:54:08.838052 29125 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:08.838109 29125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:08.838331 29125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:08.840327 29125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-1/data/instance:
uuid: "1e4468075be24daca59063ea16513474"
format_stamp: "Formatted at 2026-09-28 23:54:08 on dist-test-slave-p5bw"
server_key: "a4b975f61a4fd20dbf640a897770b9af"
server_key_iv: "e001dde2ca4a6d41fc0ce4718941e062"
server_key_version: "encryptionkey@0"
I20260928 23:54:08.840945 29125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-1/wal/instance:
uuid: "1e4468075be24daca59063ea16513474"
format_stamp: "Formatted at 2026-09-28 23:54:08 on dist-test-slave-p5bw"
server_key: "a4b975f61a4fd20dbf640a897770b9af"
server_key_iv: "e001dde2ca4a6d41fc0ce4718941e062"
server_key_version: "encryptionkey@0"
I20260928 23:54:08.844347 29125 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 23:54:08.846611 29140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:08.847718 29125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:54:08.847841 29125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-1/wal
uuid: "1e4468075be24daca59063ea16513474"
format_stamp: "Formatted at 2026-09-28 23:54:08 on dist-test-slave-p5bw"
server_key: "a4b975f61a4fd20dbf640a897770b9af"
server_key_iv: "e001dde2ca4a6d41fc0ce4718941e062"
server_key_version: "encryptionkey@0"
I20260928 23:54:08.847944 29125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:08.869836 29125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:08.873054 29125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:08.873256 29125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:08.879864 29125 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.61:38531
I20260928 23:54:08.880152 29192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.61:38531 every 8 connection(s)
I20260928 23:54:08.880887 29125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-1/data/info.pb
I20260928 23:54:08.883436 29193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:08.887049 29193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41363 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:54:08.890054 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 29125
I20260928 23:54:08.890183 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-1/wal/instance
I20260928 23:54:08.890465 6752 external_mini_cluster.cc:1428] Setting key 8e935fdc3065f827954e20a35d5a9385
Sep 28 23:54:08 dist-test-slave-p5bw krb5kdc[29037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639648, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.61@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.152.60@KRBTEST.COM; defaulting to no policy
I20260928 23:54:08.899935 29122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:08.888798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45878 (local address 127.6.152.62:41363)
0928 23:54:08.888975 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:08.888978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:08.889690 (+ 712us) server_negotiation.cc:407] Connection header received
0928 23:54:08.890711 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:08.890714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:08.890775 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:08.890873 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:08.891958 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:08.892491 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:08.893658 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:08.893897 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:08.894010 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:08.896313 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:08.896321 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:08.896324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:08.896345 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:08.897909 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:08.898390 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:08.898425 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:08.898427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:08.898506 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:08.898828 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:08.898833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:08.898835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:08.899011 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:08.899116 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:08.899646 (+ 530us) server_negotiation.cc:299] Negotiation successful
0928 23:54:08.899766 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.60@KRBTEST.COM" created.
I20260928 23:54:08.900695 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:08.889142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:41363 (local address 127.0.0.1:45878)
0928 23:54:08.889544 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:08.889585 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:08.890484 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:08.891060 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:08.891066 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:08.891382 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:08.891815 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:08.891827 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:08.892617 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:08.892621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:08.893532 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:08.893542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:08.894044 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:08.894048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:08.894220 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:08.894797 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:08.894813 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:08.896155 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:08.898023 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:08.898028 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:08.898040 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:08.898285 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:08.898598 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:08.898602 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:08.898604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:08.898734 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:08.899113 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:08.899120 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:08.899352 (+ 232us) client_negotiation.cc:769] Sending connection context
0928 23:54:08.899581 (+ 229us) client_negotiation.cc:240] Negotiation successful
0928 23:54:08.899884 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":103,"threads_started":1}
I20260928 23:54:08.902103 29193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.61" port: 38531 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:54:08 dist-test-slave-p5bw krb5kdc[29037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639648, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.61@KRBTEST.COM for kudu/127.6.152.61@KRBTEST.COM
I20260928 23:54:08.910982 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:08.902563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:38531 (local address 127.0.0.1:39602)
0928 23:54:08.902684 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:08.902696 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:08.902758 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:08.903301 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:08.903304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:08.903525 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:08.903625 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:08.903631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:08.904601 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:08.904607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:08.905208 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:08.905224 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:08.905796 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:08.905799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:08.905883 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:08.906339 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:08.906360 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:08.907729 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:08.909508 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:08.909514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:08.909519 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:08.909783 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:08.910038 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:08.910040 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:08.910041 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:08.910113 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:08.910560 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:08.910562 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:08.910631 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 23:54:08.910708 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 23:54:08.910832 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 23:54:08.911118 29198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:08.902729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39602 (local address 127.6.152.61:38531)
0928 23:54:08.902993 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:08.903000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:08.903016 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:54:08.903042 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:08.903046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:08.903174 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:08.903322 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:08.903777 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:08.904479 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:08.905327 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:08.905670 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:08.905790 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:08.907860 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:08.907868 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:08.907877 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:08.907899 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:08.909356 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:08.909883 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:08.909887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:08.909888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:08.909959 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:08.910225 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:08.910231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:08.910236 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:08.910473 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:08.910573 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:08.910812 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 23:54:08.910940 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":124,"threads_started":1}
I20260928 23:54:08.912674 29193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:54:08.913409 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:08.913214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:42263 (local address 127.0.0.1:54586)
0928 23:54:08.913291 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:08.913353 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:42263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 23:54:08.913604 29144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.152.60:42263: connect: Connection refused (error 111)
I20260928 23:54:08.914471 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:08.914286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:42263 (local address 127.0.0.1:54594)
0928 23:54:08.914379 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:08.914426 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:42263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 23:54:08.914644 29193 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.60:42263: Network error: Client connection negotiation failed: client connection to 127.6.152.60:42263: connect: Connection refused (error 111)
I20260928 23:54:08.914896 29118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.61" port: 38531 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.60.keytab.
Entry for principal kudu/127.6.152.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.60.keytab.
I20260928 23:54:08.916088 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.60:42263
--webserver_interface=127.6.152.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.60.keytab
--principal=kudu/127.6.152.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:41863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:41363,127.6.152.61:38531,127.6.152.60:42263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:54:08 dist-test-slave-p5bw krb5kdc[29037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639648, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for kudu/127.6.152.61@KRBTEST.COM
I20260928 23:54:08.924244 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:08.915331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:38531 (local address 127.0.0.1:39612)
0928 23:54:08.915447 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:08.915461 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:08.915540 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:08.915757 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:08.915771 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:08.915907 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:08.915996 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:08.916000 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:08.917022 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:08.917024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:08.918478 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:08.918516 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:08.918972 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:08.918979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:08.919126 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:08.919555 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:08.919574 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:08.920951 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:08.922880 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:08.922882 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:08.922884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:08.923063 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:08.923372 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:08.923374 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:08.923376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:08.923452 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:08.923893 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:08.923896 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:08.923969 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 23:54:08.924046 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 23:54:08.924137 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 23:54:08.924471 29198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:08.915379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39612 (local address 127.6.152.61:38531)
0928 23:54:08.915488 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:08.915492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:08.915509 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:54:08.915603 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:08.915606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:08.915653 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:08.915734 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:08.916115 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:08.916920 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:08.918636 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:08.918857 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:08.918935 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:08.921085 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:08.921092 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:08.921096 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:08.921127 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:08.922773 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:08.923162 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:08.923168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:08.923171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:08.923275 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:08.923543 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:08.923549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:08.923552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:08.923799 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:08.923911 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:08.924134 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 23:54:08.924260 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 23:54:08.925630 29118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:54:08.926316 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:08.926080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:42263 (local address 127.0.0.1:54610)
0928 23:54:08.926188 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:08.926244 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:42263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 23:54:08.927011 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:08.926788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:42263 (local address 127.0.0.1:54616)
0928 23:54:08.926927 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:08.926967 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:42263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260928 23:54:08.927211 29118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.60:42263: Network error: Client connection negotiation failed: client connection to 127.6.152.60:42263: connect: Connection refused (error 111)
I20260928 23:54:08.958382 29118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } attempt: 1
I20260928 23:54:08.959228 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:08.958938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:42263 (local address 127.0.0.1:54620)
0928 23:54:08.959091 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:08.959153 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:42263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 23:54:08.959944 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:08.959724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:42263 (local address 127.0.0.1:54632)
0928 23:54:08.959832 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:08.959885 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:42263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 23:54:08.960073 29118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.60:42263: Network error: Client connection negotiation failed: client connection to 127.6.152.60:42263: connect: Connection refused (error 111)
I20260928 23:54:08.963858 29193 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } attempt: 1
I20260928 23:54:08.964529 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:08.964291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:42263 (local address 127.0.0.1:54644)
0928 23:54:08.964419 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:08.964467 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:42263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 23:54:08.965243 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:08.965041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:42263 (local address 127.0.0.1:54648)
0928 23:54:08.965143 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:08.965188 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:42263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 23:54:08.965363 29193 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.60:42263: Network error: Client connection negotiation failed: client connection to 127.6.152.60:42263: connect: Connection refused (error 111)
W20260928 23:54:09.027027 29200 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:54:09.027375 29200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:09.027450 29200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:09.027520 29200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:09.031556 29200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:54:09.031641 29200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:09.031677 29200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:09.031747 29200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:54:09.031791 29200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:54:09.033533 29193 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } attempt: 2
I20260928 23:54:09.034339 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:09.034047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:42263 (local address 127.0.0.1:54660)
0928 23:54:09.034190 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:09.034253 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:42263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 23:54:09.035037 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:09.034825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:42263 (local address 127.0.0.1:54674)
0928 23:54:09.034935 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:09.034985 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:42263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 23:54:09.035166 29193 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.60:42263: Network error: Client connection negotiation failed: client connection to 127.6.152.60:42263: connect: Connection refused (error 111)
I20260928 23:54:09.035234 29118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } attempt: 2
I20260928 23:54:09.035902 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:09.035652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:42263 (local address 127.0.0.1:54690)
0928 23:54:09.035790 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:09.035839 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:42263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 23:54:09.036620 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:09.036436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:42263 (local address 127.0.0.1:54698)
0928 23:54:09.036526 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:09.036569 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:42263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 23:54:09.036834 29118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.60:42263: Network error: Client connection negotiation failed: client connection to 127.6.152.60:42263: connect: Connection refused (error 111)
I20260928 23:54:09.037789 29200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:41863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:41363,127.6.152.61:38531,127.6.152.60:42263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.60.keytab
--principal=kudu/127.6.152.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.60:42263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.152.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:09.038820 29200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:09.039789 29200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:09.048051 29208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:09.048010 29205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:09.047987 29206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:09.048018 29200 server_base.cc:1034] running on GCE node
I20260928 23:54:09.048741 29200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:09.049922 29200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:09.051119 29200 hybrid_clock.cc:648] HybridClock initialized: now 1790639649051106 us; error 30 us; skew 500 ppm
Sep 28 23:54:09 dist-test-slave-p5bw krb5kdc[29037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639649, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:09.053972 29200 init.cc:377] Logged in from keytab as kudu/127.6.152.60@KRBTEST.COM (short username kudu)
I20260928 23:54:09.054828 29200 webserver.cc:460] Webserver started at http://127.6.152.60:37383/ using document root <none> and password file <none>
I20260928 23:54:09.055382 29200 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:09.055470 29200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:09.055696 29200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:09.057737 29200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-2/data/instance:
uuid: "f67acb1c9c594908b95a1b0314e5cf44"
format_stamp: "Formatted at 2026-09-28 23:54:09 on dist-test-slave-p5bw"
server_key: "ac1400175149f4b8a3d92d829d0b7f1c"
server_key_iv: "2d9df04e952f4728bf8db603a46bea89"
server_key_version: "encryptionkey@0"
I20260928 23:54:09.058303 29200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-2/wal/instance:
uuid: "f67acb1c9c594908b95a1b0314e5cf44"
format_stamp: "Formatted at 2026-09-28 23:54:09 on dist-test-slave-p5bw"
server_key: "ac1400175149f4b8a3d92d829d0b7f1c"
server_key_iv: "2d9df04e952f4728bf8db603a46bea89"
server_key_version: "encryptionkey@0"
I20260928 23:54:09.061898 29200 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260928 23:54:09.064260 29215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:09.065689 29200 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 23:54:09.065830 29200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-2/wal
uuid: "f67acb1c9c594908b95a1b0314e5cf44"
format_stamp: "Formatted at 2026-09-28 23:54:09 on dist-test-slave-p5bw"
server_key: "ac1400175149f4b8a3d92d829d0b7f1c"
server_key_iv: "2d9df04e952f4728bf8db603a46bea89"
server_key_version: "encryptionkey@0"
I20260928 23:54:09.065948 29200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:09.091996 29200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:09.095432 29200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:09.095642 29200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:09.102488 29200 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.60:42263
I20260928 23:54:09.102881 29267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.60:42263 every 8 connection(s)
I20260928 23:54:09.103544 29200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-2/data/info.pb
I20260928 23:54:09.106278 29268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:09.109889 29268 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41363 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:54:09.113101 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 29200
I20260928 23:54:09.113229 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639374546653-6752-0/minicluster-data/master-2/wal/instance
I20260928 23:54:09.113566 6752 external_mini_cluster.cc:1428] Setting key 863e2a3d7b63de9289f307a8b7215536
Sep 28 23:54:09 dist-test-slave-p5bw krb5kdc[29037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639649, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.60@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
Sep 28 23:54:09 dist-test-slave-p5bw krb5kdc[29037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.60@KRBTEST.COM
I20260928 23:54:09.123612 29122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:09.111588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45884 (local address 127.6.152.62:41363)
0928 23:54:09.111727 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:09.111731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:09.112493 (+ 762us) server_negotiation.cc:407] Connection header received
0928 23:54:09.113766 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:09.113769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:09.113816 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:09.113895 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:09.115338 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.115867 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.116927 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.117136 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.117220 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:09.119895 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:09.119899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:09.119901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:09.119925 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:09.121509 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:09.122013 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:09.122017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:09.122019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:09.122102 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:09.122451 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:09.122456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:09.122458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:09.122691 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:09.122795 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:09.123369 (+ 574us) server_negotiation.cc:299] Negotiation successful
0928 23:54:09.123491 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:54:09.124593 29272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:09.114269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54704 (local address 127.6.152.60:42263)
0928 23:54:09.114649 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:09.114657 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:09.114677 (+ 20us) server_negotiation.cc:407] Connection header received
0928 23:54:09.114788 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:09.114792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:09.114947 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:09.115074 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:09.115870 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.116714 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.117794 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.118249 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.118329 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:09.120999 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:09.121008 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:09.121023 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:09.121054 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:09.122571 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:09.123010 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:09.123015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:09.123017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:09.123095 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:09.123406 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:09.123409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:09.123410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:09.123677 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:09.123860 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:09.124207 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 23:54:09.124392 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":156,"threads_started":1}
I20260928 23:54:09.124603 29270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:09.111934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:41363 (local address 127.0.0.1:45884)
0928 23:54:09.112354 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:09.112395 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:09.113496 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:09.114069 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:09.114097 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:09.114551 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:09.115182 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:09.115204 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.115981 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.115984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:09.116752 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:09.116794 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.117286 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.117290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:09.117478 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:09.118115 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:09.118131 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:09.119741 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:09.121626 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:09.121636 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:09.121645 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:09.121906 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:09.122198 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:09.122202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:09.122204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:09.122351 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:09.122806 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:09.122811 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:09.123082 (+ 271us) client_negotiation.cc:769] Sending connection context
0928 23:54:09.123298 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 23:54:09.123607 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":108,"threads_started":1}
I20260928 23:54:09.125850 29268 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.61" port: 38531 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:54:09.126331 29193 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } attempt: 3
Sep 28 23:54:09 dist-test-slave-p5bw krb5kdc[29037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639649, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.60@KRBTEST.COM for kudu/127.6.152.61@KRBTEST.COM
Sep 28 23:54:09 dist-test-slave-p5bw krb5kdc[29037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639648, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.61@KRBTEST.COM for kudu/127.6.152.60@KRBTEST.COM
Sep 28 23:54:09 dist-test-slave-p5bw krb5kdc[29037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:54:09.135324 29198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:09.126481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39616 (local address 127.6.152.61:38531)
0928 23:54:09.126621 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:09.126626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:09.126640 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:54:09.126729 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:09.126735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:09.126796 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:09.126855 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:09.127431 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.128110 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.129095 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.129311 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.129428 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:09.131573 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:09.131577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:09.131581 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:09.131612 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:09.133259 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:09.133805 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:09.133809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:09.133811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:09.133901 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:09.134231 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:09.134235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:09.134240 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:09.134548 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:09.134624 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:09.135001 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 23:54:09.135133 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51,"spinlock_wait_cycles":5248}
I20260928 23:54:09.135784 29270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:09.126334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:38531 (local address 127.0.0.1:39616)
0928 23:54:09.126539 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:09.126553 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:09.126622 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:09.127021 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:09.127026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:09.127195 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:09.127303 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:09.127312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.128218 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.128221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:09.128990 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:09.128998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.129423 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.129427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:09.129521 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:09.129946 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:09.129960 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:09.131431 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:09.133387 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:09.133391 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:09.133394 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:09.133634 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:09.134009 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:09.134013 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:09.134016 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:09.134109 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:09.134701 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:09.134706 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:09.134802 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 23:54:09.135549 (+ 747us) client_negotiation.cc:240] Negotiation successful
0928 23:54:09.135681 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 23:54:09.135978 29118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } attempt: 3
I20260928 23:54:09.136073 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:09.126691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:42263 (local address 127.0.0.1:54714)
0928 23:54:09.126786 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:09.126800 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:09.126888 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:09.127551 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:09.127554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:09.127730 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:09.127809 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:09.127815 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.128784 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.128787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:09.129610 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:09.129619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.130288 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.130292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:09.130375 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:09.130856 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:09.130873 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:09.132179 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:09.134468 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:09.134474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:09.134478 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:09.134722 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:09.135023 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:09.135026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:09.135028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:09.135114 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:09.135587 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:09.135590 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:09.135683 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 23:54:09.135782 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 23:54:09.135914 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"spinlock_wait_cycles":2688}
I20260928 23:54:09.136569 29268 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:54:09.136756 29272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:09.127275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54714 (local address 127.6.152.60:42263)
0928 23:54:09.127383 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:09.127389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:09.127401 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:54:09.127431 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:09.127434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:09.127475 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:09.127530 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:09.127943 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.128638 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.129941 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.130170 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.130291 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:09.132327 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:09.132332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:09.132338 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:09.132370 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:09.134361 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:09.134825 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:09.134828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:09.134830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:09.134917 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:09.135239 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:09.135243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:09.135244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:09.135468 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:09.136340 (+ 872us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:09.136436 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 23:54:09.136549 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 23:54:09.139199 29122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:09.127157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45900 (local address 127.6.152.62:41363)
0928 23:54:09.127254 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:09.127257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:09.127266 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:54:09.127300 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:09.127302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:09.127333 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:09.127421 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:09.128736 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.129507 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.130648 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.130949 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.131007 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:09.134223 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:09.134226 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:09.134229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:09.134252 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:09.137028 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:09.137649 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:09.137657 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:09.137662 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:09.137753 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:09.138080 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:09.138084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:09.138086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:09.138320 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:09.138555 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:09.138943 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 23:54:09.139058 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 23:54:09.140105 29193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e4468075be24daca59063ea16513474: Bootstrap starting.
Sep 28 23:54:09 dist-test-slave-p5bw krb5kdc[29037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639648, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for kudu/127.6.152.60@KRBTEST.COM
Sep 28 23:54:09 dist-test-slave-p5bw krb5kdc[29037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639649, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.60@KRBTEST.COM for kudu/127.6.152.60@KRBTEST.COM
I20260928 23:54:09.143646 29193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e4468075be24daca59063ea16513474: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:09.144593 29193 log.cc:824] T 00000000000000000000000000000000 P 1e4468075be24daca59063ea16513474: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:09.146684 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:09.136543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:42263 (local address 127.0.0.1:54726)
0928 23:54:09.136862 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:09.136878 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:09.136951 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:09.137707 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:09.137710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:09.137891 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:09.138006 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:09.138016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.138989 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.138995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:09.139719 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:09.139729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.140407 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.140414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:09.140501 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:09.140984 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:09.141001 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:09.142849 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:09.145112 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:09.145115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:09.145117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:09.145374 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:09.145714 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:09.145718 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:09.145723 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:09.145827 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:09.146239 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:09.146241 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:09.146328 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 23:54:09.146414 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 23:54:09.146538 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214}
I20260928 23:54:09.147136 29193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e4468075be24daca59063ea16513474: No bootstrap required, opened a new log
I20260928 23:54:09.147518 29273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:09.137321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54732 (local address 127.6.152.60:42263)
0928 23:54:09.137422 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:09.137431 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:09.137472 (+ 41us) server_negotiation.cc:407] Connection header received
0928 23:54:09.137588 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:09.137591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:09.137632 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:09.137705 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:09.138202 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.139344 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.140453 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.140779 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.140928 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:09.143368 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:09.143372 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:09.143374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:09.143396 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:09.145478 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:09.145915 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:09.145918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:09.145920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:09.145984 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:09.146196 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:09.146199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:09.146200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:09.146388 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:09.146466 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:09.147022 (+ 556us) server_negotiation.cc:299] Negotiation successful
0928 23:54:09.147326 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":157,"server-negotiator.queue_time_us":42}
I20260928 23:54:09.147560 29272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:09.136558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54726 (local address 127.6.152.60:42263)
0928 23:54:09.137115 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:09.137120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:09.137131 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:54:09.137161 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:09.137163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:09.137200 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:09.138126 (+ 926us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:09.138207 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.138860 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.139878 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.140285 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.140385 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:09.142981 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:09.142985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:09.142987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:09.143011 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:09.144987 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:09.145506 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:09.145512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:09.145517 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:09.145604 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:09.145942 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:09.145945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:09.145946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:09.146134 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:09.147149 (+ 1015us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:09.147230 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 23:54:09.147391 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":493,"thread_start_us":100,"threads_started":1}
I20260928 23:54:09.147954 29270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:09.137120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:42263 (local address 127.0.0.1:54732)
0928 23:54:09.137416 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:09.137429 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:09.137507 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:09.137819 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:09.137822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:09.138001 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:09.138097 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:09.138103 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.139465 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.139467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:09.140320 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:09.140329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.141028 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.141034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:09.141123 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:09.141490 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:09.141508 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:09.143239 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:09.145576 (+ 2337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:09.145579 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:09.145581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:09.145809 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:09.146058 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:09.146061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:09.146063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:09.146122 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:09.146493 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:09.146496 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:09.146560 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 23:54:09.146640 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 23:54:09.147152 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":74,"spinlock_wait_cycles":4352}
Sep 28 23:54:09 dist-test-slave-p5bw krb5kdc[29037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.61@KRBTEST.COM
I20260928 23:54:09.150645 29118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab: Bootstrap starting.
I20260928 23:54:09.150950 29193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e4468075be24daca59063ea16513474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b408b7f6c42a297d449133d8b73ab" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41363 } } peers { permanent_uuid: "1e4468075be24daca59063ea16513474" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 38531 } } peers { permanent_uuid: "f67acb1c9c594908b95a1b0314e5cf44" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } }
I20260928 23:54:09.151137 29193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e4468075be24daca59063ea16513474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4468075be24daca59063ea16513474, State: Initialized, Role: FOLLOWER
I20260928 23:54:09.151523 29193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e4468075be24daca59063ea16513474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b408b7f6c42a297d449133d8b73ab" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41363 } } peers { permanent_uuid: "1e4468075be24daca59063ea16513474" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 38531 } } peers { permanent_uuid: "f67acb1c9c594908b95a1b0314e5cf44" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } }
I20260928 23:54:09.152041 29275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e4468075be24daca59063ea16513474 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b408b7f6c42a297d449133d8b73ab" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41363 } } peers { permanent_uuid: "1e4468075be24daca59063ea16513474" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 38531 } } peers { permanent_uuid: "f67acb1c9c594908b95a1b0314e5cf44" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } } }
I20260928 23:54:09.152179 29275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e4468075be24daca59063ea16513474 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:54:09.152854 29193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e4468075be24daca59063ea16513474 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:54:09.153376 29118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:09.154217 29118 log.cc:824] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:09.155819 29268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67acb1c9c594908b95a1b0314e5cf44: Bootstrap starting.
I20260928 23:54:09.156226 29118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab: No bootstrap required, opened a new log
W20260928 23:54:09.157757 29287 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e4468075be24daca59063ea16513474: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:54:09.157851 29287 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:54:09.158347 29268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f67acb1c9c594908b95a1b0314e5cf44: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:09.159174 29268 log.cc:824] T 00000000000000000000000000000000 P f67acb1c9c594908b95a1b0314e5cf44: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:09.161111 29118 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b408b7f6c42a297d449133d8b73ab" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41363 } } peers { permanent_uuid: "1e4468075be24daca59063ea16513474" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 38531 } } peers { permanent_uuid: "f67acb1c9c594908b95a1b0314e5cf44" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } }
I20260928 23:54:09.161401 29118 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8b408b7f6c42a297d449133d8b73ab, State: Initialized, Role: FOLLOWER
I20260928 23:54:09.161507 29268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67acb1c9c594908b95a1b0314e5cf44: No bootstrap required, opened a new log
I20260928 23:54:09.161856 29118 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b408b7f6c42a297d449133d8b73ab" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41363 } } peers { permanent_uuid: "1e4468075be24daca59063ea16513474" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 38531 } } peers { permanent_uuid: "f67acb1c9c594908b95a1b0314e5cf44" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } }
I20260928 23:54:09.163266 29118 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:54:09.165553 29268 raft_consensus.cc:348] T 00000000000000000000000000000000 P f67acb1c9c594908b95a1b0314e5cf44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b408b7f6c42a297d449133d8b73ab" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41363 } } peers { permanent_uuid: "1e4468075be24daca59063ea16513474" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 38531 } } peers { permanent_uuid: "f67acb1c9c594908b95a1b0314e5cf44" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } }
I20260928 23:54:09.165804 29268 raft_consensus.cc:729] T 00000000000000000000000000000000 P f67acb1c9c594908b95a1b0314e5cf44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f67acb1c9c594908b95a1b0314e5cf44, State: Initialized, Role: FOLLOWER
I20260928 23:54:09.162209 29125 master_runner.cc:186] Error getting master registration for 127.6.152.62:41363: OK, instance_id { permanent_uuid: "ae8b408b7f6c42a297d449133d8b73ab" instance_seqno: 1790639648671750 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:54:09.167663 29125 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:54:09.166137 29268 consensus_queue.cc:260] T 00000000000000000000000000000000 P f67acb1c9c594908b95a1b0314e5cf44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b408b7f6c42a297d449133d8b73ab" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41363 } } peers { permanent_uuid: "1e4468075be24daca59063ea16513474" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 38531 } } peers { permanent_uuid: "f67acb1c9c594908b95a1b0314e5cf44" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } }
W20260928 23:54:09.168768 29071 tablet.cc:2369] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:54:09.169096 29301 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:54:09.169217 29301 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:54:09.169451 29198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:09.142263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39622 (local address 127.6.152.61:38531)
0928 23:54:09.142358 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:09.142362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:09.142508 (+ 146us) server_negotiation.cc:407] Connection header received
0928 23:54:09.142684 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:09.142687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:09.142728 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:09.142788 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:09.143963 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.145227 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.146351 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.146684 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.146762 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:09.159772 (+ 13010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:09.159776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:09.159779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:09.159807 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:09.166036 (+ 6229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:09.166684 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:09.166687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:09.166689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:09.166779 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:09.167992 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:09.167996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:09.167997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:09.168226 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:09.168420 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:09.169173 (+ 753us) server_negotiation.cc:299] Negotiation successful
0928 23:54:09.169287 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 23:54:09.169836 29268 sys_catalog.cc:564] T 00000000000000000000000000000000 P f67acb1c9c594908b95a1b0314e5cf44 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:54:09.170048 29290 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b408b7f6c42a297d449133d8b73ab" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41363 } } peers { permanent_uuid: "1e4468075be24daca59063ea16513474" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 38531 } } peers { permanent_uuid: "f67acb1c9c594908b95a1b0314e5cf44" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } } }
I20260928 23:54:09.170168 29290 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:54:09.172008 29303 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67acb1c9c594908b95a1b0314e5cf44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b408b7f6c42a297d449133d8b73ab" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41363 } } peers { permanent_uuid: "1e4468075be24daca59063ea16513474" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 38531 } } peers { permanent_uuid: "f67acb1c9c594908b95a1b0314e5cf44" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } } }
I20260928 23:54:09.172115 29303 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67acb1c9c594908b95a1b0314e5cf44 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 23:54:09.175140 29314 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f67acb1c9c594908b95a1b0314e5cf44: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:54:09.175257 29314 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:54:09.176754 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:54:09.178022 29092 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ae8b408b7f6c42a297d449133d8b73ab"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45900
I20260928 23:54:09.178225 29092 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 23:54:09.178334 29092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:09.179631 29092 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b408b7f6c42a297d449133d8b73ab" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41363 } } peers { permanent_uuid: "1e4468075be24daca59063ea16513474" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 38531 } } peers { permanent_uuid: "f67acb1c9c594908b95a1b0314e5cf44" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } }
I20260928 23:54:09.180505 29092 leader_election.cc:288] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab [CANDIDATE]: Term 1 election: Requested vote from peers 1e4468075be24daca59063ea16513474 (127.6.152.61:38531), f67acb1c9c594908b95a1b0314e5cf44 (127.6.152.60:42263)
I20260928 23:54:09.181245 29167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae8b408b7f6c42a297d449133d8b73ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1e4468075be24daca59063ea16513474"
I20260928 23:54:09.181466 29167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e4468075be24daca59063ea16513474 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:09.182710 29167 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1e4468075be24daca59063ea16513474 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae8b408b7f6c42a297d449133d8b73ab in term 1.
I20260928 23:54:09.183338 29050 master_runner.cc:263] No leader master found from master ae8b408b7f6c42a297d449133d8b73ab
I20260928 23:54:09.183403 29050 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:54:09.183712 29242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae8b408b7f6c42a297d449133d8b73ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f67acb1c9c594908b95a1b0314e5cf44"
I20260928 23:54:09.183912 29242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f67acb1c9c594908b95a1b0314e5cf44 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:09.185169 29242 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f67acb1c9c594908b95a1b0314e5cf44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae8b408b7f6c42a297d449133d8b73ab in term 1.
I20260928 23:54:09.185667 29066 leader_election.cc:302] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e4468075be24daca59063ea16513474, ae8b408b7f6c42a297d449133d8b73ab; no voters:
I20260928 23:54:09.185947 29290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:09.186170 29290 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab [term 1 LEADER]: Becoming Leader. State: Replica: ae8b408b7f6c42a297d449133d8b73ab, State: Running, Role: LEADER
I20260928 23:54:09.186506 29290 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b408b7f6c42a297d449133d8b73ab" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41363 } } peers { permanent_uuid: "1e4468075be24daca59063ea16513474" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 38531 } } peers { permanent_uuid: "f67acb1c9c594908b95a1b0314e5cf44" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } }
I20260928 23:54:09.188215 29290 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae8b408b7f6c42a297d449133d8b73ab. Latest consensus state: current_term: 1 leader_uuid: "ae8b408b7f6c42a297d449133d8b73ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b408b7f6c42a297d449133d8b73ab" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41363 } } peers { permanent_uuid: "1e4468075be24daca59063ea16513474" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 38531 } } peers { permanent_uuid: "f67acb1c9c594908b95a1b0314e5cf44" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } } }
I20260928 23:54:09.188338 29290 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:09.194026 29200 master_runner.cc:263] No leader master found from master f67acb1c9c594908b95a1b0314e5cf44
I20260928 23:54:09.194103 29200 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:54:09.195756 29318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:54:09.196688 29318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:54:09.209430 29242 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f67acb1c9c594908b95a1b0314e5cf44 [term 1 FOLLOWER]: Refusing update from remote peer ae8b408b7f6c42a297d449133d8b73ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:54:09.210095 29316 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f67acb1c9c594908b95a1b0314e5cf44" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:54:09.214236 29303 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67acb1c9c594908b95a1b0314e5cf44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae8b408b7f6c42a297d449133d8b73ab. Latest consensus state: current_term: 1 leader_uuid: "ae8b408b7f6c42a297d449133d8b73ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b408b7f6c42a297d449133d8b73ab" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41363 } } peers { permanent_uuid: "1e4468075be24daca59063ea16513474" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 38531 } } peers { permanent_uuid: "f67acb1c9c594908b95a1b0314e5cf44" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } } }
I20260928 23:54:09.214356 29303 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67acb1c9c594908b95a1b0314e5cf44 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:54:09.215211 29167 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1e4468075be24daca59063ea16513474 [term 1 FOLLOWER]: Refusing update from remote peer ae8b408b7f6c42a297d449133d8b73ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:54:09.215778 29316 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e4468075be24daca59063ea16513474" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:54:09.219602 29275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e4468075be24daca59063ea16513474 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae8b408b7f6c42a297d449133d8b73ab. Latest consensus state: current_term: 1 leader_uuid: "ae8b408b7f6c42a297d449133d8b73ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b408b7f6c42a297d449133d8b73ab" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41363 } } peers { permanent_uuid: "1e4468075be24daca59063ea16513474" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 38531 } } peers { permanent_uuid: "f67acb1c9c594908b95a1b0314e5cf44" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } } }
I20260928 23:54:09.219741 29275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e4468075be24daca59063ea16513474 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:54:09.229544 29272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:09.197554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54748 (local address 127.6.152.60:42263)
0928 23:54:09.197657 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:09.197662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:09.201832 (+ 4170us) server_negotiation.cc:407] Connection header received
0928 23:54:09.203114 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:09.203117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:09.203160 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:09.203207 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:09.206109 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.206854 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.214771 (+ 7917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.215070 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.215143 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:09.221665 (+ 6522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:09.221669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:09.221672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:09.221698 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:09.223349 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:09.226061 (+ 2712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:09.226064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:09.226066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:09.226149 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:09.226870 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:09.226873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:09.226875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:09.227087 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:09.227213 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:09.229272 (+ 2059us) server_negotiation.cc:299] Negotiation successful
0928 23:54:09.229399 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 23:54:09.231227 29122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:09.197161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45912 (local address 127.6.152.62:41363)
0928 23:54:09.197269 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:09.197276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:09.199271 (+ 1995us) server_negotiation.cc:407] Connection header received
0928 23:54:09.201908 (+ 2637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:09.201912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:09.201963 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:09.202012 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:09.204128 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.204933 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.214181 (+ 9248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.214544 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.214621 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:09.223881 (+ 9260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:09.223885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:09.223888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:09.223915 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:09.225570 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:09.226475 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:09.226479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:09.226480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:09.226563 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:09.228541 (+ 1978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:09.228544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:09.228546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:09.228769 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:09.228937 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:09.230966 (+ 2029us) server_negotiation.cc:299] Negotiation successful
0928 23:54:09.231081 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 23:54:09.232546 29290 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae8b408b7f6c42a297d449133d8b73ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b408b7f6c42a297d449133d8b73ab" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41363 } } peers { permanent_uuid: "1e4468075be24daca59063ea16513474" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 38531 } } peers { permanent_uuid: "f67acb1c9c594908b95a1b0314e5cf44" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } } }
I20260928 23:54:09.232657 29290 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:09.232739 29290 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae8b408b7f6c42a297d449133d8b73ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b408b7f6c42a297d449133d8b73ab" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41363 } } peers { permanent_uuid: "1e4468075be24daca59063ea16513474" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 38531 } } peers { permanent_uuid: "f67acb1c9c594908b95a1b0314e5cf44" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } } }
I20260928 23:54:09.232797 29290 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:09.233140 29275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e4468075be24daca59063ea16513474 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae8b408b7f6c42a297d449133d8b73ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b408b7f6c42a297d449133d8b73ab" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41363 } } peers { permanent_uuid: "1e4468075be24daca59063ea16513474" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 38531 } } peers { permanent_uuid: "f67acb1c9c594908b95a1b0314e5cf44" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } } }
I20260928 23:54:09.233258 29275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e4468075be24daca59063ea16513474 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:54:09.233577 29326 mvcc.cc:205] Tried to move back new op lower bound from 7334460003125800960 to 7334460003070652416. Current Snapshot: MvccSnapshot[applied={T|T < 7334460003125800960}]
I20260928 23:54:09.233732 29329 mvcc.cc:205] Tried to move back new op lower bound from 7334460003125800960 to 7334460003070652416. Current Snapshot: MvccSnapshot[applied={T|T < 7334460003125800960}]
I20260928 23:54:09.233932 29303 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67acb1c9c594908b95a1b0314e5cf44 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae8b408b7f6c42a297d449133d8b73ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8b408b7f6c42a297d449133d8b73ab" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 41363 } } peers { permanent_uuid: "1e4468075be24daca59063ea16513474" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 38531 } } peers { permanent_uuid: "f67acb1c9c594908b95a1b0314e5cf44" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 42263 } } }
I20260928 23:54:09.234048 29303 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67acb1c9c594908b95a1b0314e5cf44 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:54:09.234727 29327 mvcc.cc:205] Tried to move back new op lower bound from 7334460003125800960 to 7334460003070652416. Current Snapshot: MvccSnapshot[applied={T|T < 7334460003125800960}]
I20260928 23:54:09.238945 29198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:09.198185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39638 (local address 127.6.152.61:38531)
0928 23:54:09.198298 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:09.198303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:09.208142 (+ 9839us) server_negotiation.cc:407] Connection header received
0928 23:54:09.215292 (+ 7150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:09.215295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:09.215347 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:09.215411 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:09.219752 (+ 4341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.220508 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.227534 (+ 7026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:09.227843 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:09.227928 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:09.235417 (+ 7489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:09.235421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:09.235424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:09.235450 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:09.237222 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:09.237687 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:09.237690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:09.237692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:09.237767 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:09.238053 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:09.238056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:09.238058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:09.238251 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:09.238374 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:09.238694 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 23:54:09.238802 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 23:54:09.243592 29318 catalog_manager.cc:1347] Generated new cluster ID: 5eb341998eef4f2eb083a9deb1e2fe9c
I20260928 23:54:09.243657 29318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:54:09.255555 29318 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:54:09.256732 29318 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:54:09.269341 29318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae8b408b7f6c42a297d449133d8b73ab: Generated new TSK 0
I20260928 23:54:09.270222 29318 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:54:09.305933 29272 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:09.303830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54752 (local address 127.6.152.60:42263)
0928 23:54:09.303971 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:09.303977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:09.304679 (+ 702us) server_negotiation.cc:407] Connection header received
0928 23:54:09.304951 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:09.304955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:09.305008 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:09.305066 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:09.305772 (+ 706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54752: BlockingRecv error: recv got EOF from 127.0.0.1:54752 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 23:54:09.308538 29122 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:09.306692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45926 (local address 127.6.152.62:41363)
0928 23:54:09.306823 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:09.306827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:09.307528 (+ 701us) server_negotiation.cc:407] Connection header received
0928 23:54:09.307716 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:09.307720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:09.307770 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:09.307838 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:09.308403 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45926: BlockingRecv error: recv got EOF from 127.0.0.1:45926 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 23:54:09.309912 29198 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:09.307266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39642 (local address 127.6.152.61:38531)
0928 23:54:09.307417 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:09.307422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:09.308981 (+ 1559us) server_negotiation.cc:407] Connection header received
0928 23:54:09.309161 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:09.309164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:09.309217 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:09.309271 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:09.309763 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39642: BlockingRecv error: recv got EOF from 127.0.0.1:39642 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 23:54:09.313975 29198 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:09.312410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39658 (local address 127.6.152.61:38531)
0928 23:54:09.312512 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:09.312516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:09.313005 (+ 489us) server_negotiation.cc:407] Connection header received
0928 23:54:09.313223 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:09.313226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:09.313301 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:09.313357 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:09.313873 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39658: BlockingRecv error: recv got EOF from 127.0.0.1:39658 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 23:54:09.319078 29198 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:09.317605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39660 (local address 127.6.152.61:38531)
0928 23:54:09.317702 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:09.317706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:09.318144 (+ 438us) server_negotiation.cc:407] Connection header received
0928 23:54:09.318342 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:09.318346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:09.318397 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:09.318451 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:09.318980 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39660: BlockingRecv error: recv got EOF from 127.0.0.1:39660 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 23:54:09.324873 29198 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:09.323338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39676 (local address 127.6.152.61:38531)
0928 23:54:09.323440 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:09.323445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:09.323945 (+ 500us) server_negotiation.cc:407] Connection header received
0928 23:54:09.324128 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:09.324132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:09.324181 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:09.324233 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:09.324734 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39676: BlockingRecv error: recv got EOF from 127.0.0.1:39676 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 23:54:09.333187 29198 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:09.331303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.6.152.61:38531)
0928 23:54:09.331463 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:09.331468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:09.332052 (+ 584us) server_negotiation.cc:407] Connection header received
0928 23:54:09.332284 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:09.332288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:09.332346 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:09.332404 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:09.333072 (+ 668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39684: BlockingRecv error: recv got EOF from 127.0.0.1:39684 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260928 23:54:09.346287 29198 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:09.344161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39694 (local address 127.6.152.61:38531)
0928 23:54:09.344306 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:09.344312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:09.345082 (+ 770us) server_negotiation.cc:407] Connection header received
0928 23:54:09.345331 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:09.345335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:09.345398 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:09.345462 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:09.346151 (+ 689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39694: BlockingRecv error: recv got EOF from 127.0.0.1:39694 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 23:54:09.367270 29198 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:09.365367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39710 (local address 127.6.152.61:38531)
0928 23:54:09.365505 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:09.365510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:09.366140 (+ 630us) server_negotiation.cc:407] Connection header received
0928 23:54:09.366379 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:09.366382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:09.366443 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:09.366501 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:09.367140 (+ 639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39710: BlockingRecv error: recv got EOF from 127.0.0.1:39710 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 23:54:09.403030 29198 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:09.401872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39716 (local address 127.6.152.61:38531)
0928 23:54:09.402020 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:09.402026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:09.402172 (+ 146us) server_negotiation.cc:407] Connection header received
0928 23:54:09.402281 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:09.402284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:09.402336 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:09.402409 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:09.402871 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39716: BlockingRecv error: recv got EOF from 127.0.0.1:39716 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 23:54:09.470482 29198 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:09.469074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39728 (local address 127.6.152.61:38531)
0928 23:54:09.469218 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:09.469223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:09.469444 (+ 221us) server_negotiation.cc:407] Connection header received
0928 23:54:09.469631 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:09.469634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:09.469683 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:09.469775 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:09.470328 (+ 553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39728: BlockingRecv error: recv got EOF from 127.0.0.1:39728 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 23:54:09.601907 29198 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:09.600693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39730 (local address 127.6.152.61:38531)
0928 23:54:09.600879 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:09.600889 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:09.601034 (+ 145us) server_negotiation.cc:407] Connection header received
0928 23:54:09.601157 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:09.601160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:09.601212 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:09.601291 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:09.601737 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39730: BlockingRecv error: recv got EOF from 127.0.0.1:39730 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260928 23:54:09.861115 29198 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:09.859910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39740 (local address 127.6.152.61:38531)
0928 23:54:09.860091 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:09.860096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:09.860180 (+ 84us) server_negotiation.cc:407] Connection header received
0928 23:54:09.860315 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:09.860318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:09.860371 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:09.860452 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:09.860956 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39740: BlockingRecv error: recv got EOF from 127.0.0.1:39740 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 23:54:10.159258 29287 catalog_manager.cc:1259] Loaded cluster ID: 5eb341998eef4f2eb083a9deb1e2fe9c
I20260928 23:54:10.159353 29287 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1e4468075be24daca59063ea16513474: loading cluster ID for follower catalog manager: success
I20260928 23:54:10.161242 29287 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1e4468075be24daca59063ea16513474: acquiring CA information for follower catalog manager: success
I20260928 23:54:10.162500 29287 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1e4468075be24daca59063ea16513474: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:54:10.176433 29314 catalog_manager.cc:1259] Loaded cluster ID: 5eb341998eef4f2eb083a9deb1e2fe9c
I20260928 23:54:10.176506 29314 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f67acb1c9c594908b95a1b0314e5cf44: loading cluster ID for follower catalog manager: success
I20260928 23:54:10.178085 29314 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f67acb1c9c594908b95a1b0314e5cf44: acquiring CA information for follower catalog manager: success
I20260928 23:54:10.179160 29314 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f67acb1c9c594908b95a1b0314e5cf44: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:54:10.379045 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:10.374992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39750 (local address 127.6.152.61:38531)
0928 23:54:10.375230 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:10.375238 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:10.375339 (+ 101us) server_negotiation.cc:407] Connection header received
0928 23:54:10.375527 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:10.375531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:10.375653 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:10.375773 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:54:10.376039 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:10.376574 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:10.377530 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:10.377762 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:10.377873 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:10.378206 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:54:10.378644 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:54:10.378786 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:10.378826 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 23:54:10.378880 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":67,"threads_started":1}
I20260928 23:54:10.386118 29404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:10.382257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54760 (local address 127.6.152.60:42263)
0928 23:54:10.382471 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:10.382475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:10.382488 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:54:10.382650 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:10.382652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:10.382774 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:10.382872 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:54:10.383167 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:10.383757 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:10.384637 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:10.384881 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:10.384962 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:10.385311 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:54:10.385747 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:54:10.385849 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:10.385956 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 23:54:10.386004 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":76,"threads_started":1}
I20260928 23:54:10.393605 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:10.389172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45934 (local address 127.6.152.62:41363)
0928 23:54:10.389496 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:10.389500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:10.389969 (+ 469us) server_negotiation.cc:407] Connection header received
0928 23:54:10.390071 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:10.390075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:10.390239 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:10.390384 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:54:10.390619 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:10.391149 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:10.392167 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:10.392417 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:10.392526 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:10.392884 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:54:10.393160 (+ 276us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:54:10.393266 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:10.393358 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 23:54:10.393415 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":70,"threads_started":1}
I20260928 23:54:10.393996 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:10.389664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39762 (local address 127.6.152.61:38531)
0928 23:54:10.389780 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:10.389786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:10.389904 (+ 118us) server_negotiation.cc:407] Connection header received
0928 23:54:10.390039 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:10.390042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:10.390099 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:10.390167 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:54:10.390537 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:10.391275 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:10.392549 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:10.392952 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:10.393021 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:10.393527 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:54:10.393602 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:54:10.393648 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:10.393830 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 23:54:10.393880 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 23:54:10.393996 29404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:10.389200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54762 (local address 127.6.152.60:42263)
0928 23:54:10.389318 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:10.389324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:10.389835 (+ 511us) server_negotiation.cc:407] Connection header received
0928 23:54:10.389951 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:10.389957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:10.390012 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:10.390080 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:54:10.390316 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:10.391102 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:10.392549 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:10.392913 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:10.392998 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:10.393527 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:54:10.393602 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:54:10.393648 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:10.393819 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 23:54:10.393872 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 23:54:10.396986 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 29050
I20260928 23:54:10.403199 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 29125
I20260928 23:54:10.408879 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 29200
2026-09-28T23:54:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3102 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:54:10 dist-test-slave-p5bw krb5kdc[29415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:54:10 dist-test-slave-p5bw krb5kdc[29415](info): set up 2 sockets
Sep 28 23:54:10 dist-test-slave-p5bw krb5kdc[29415](info): commencing operation
krb5kdc: starting...
W20260928 23:54:11.446345 6752 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/krb5kdc/test-user.keytab.
Sep 28 23:54:11 dist-test-slave-p5bw krb5kdc[29415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:54:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:54:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
Entry for principal kudu/127.6.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab.
I20260928 23:54:11.564350 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:45089
--webserver_interface=127.6.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:46645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:45089,127.6.152.61:37487,127.6.152.60:41187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:54:11.671866 29428 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:54:11.672181 29428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:11.672241 29428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:11.672318 29428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:54:11.676344 29428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:54:11.676436 29428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:11.676473 29428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:11.676509 29428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:54:11.676540 29428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:54:11.681799 29428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:46645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:45089,127.6.152.61:37487,127.6.152.60:41187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.62.keytab
--principal=kudu/127.6.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.62:45089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.152.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:11.682818 29428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:11.683821 29428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:11.692081 29433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:11.692091 29434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:11.692155 29436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:11.692103 29428 server_base.cc:1034] running on GCE node
I20260928 23:54:11.693532 29428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:11.694612 29428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:11.695770 29428 hybrid_clock.cc:648] HybridClock initialized: now 1790639651695746 us; error 36 us; skew 500 ppm
Sep 28 23:54:11 dist-test-slave-p5bw krb5kdc[29415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639651, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:11.702071 29428 init.cc:377] Logged in from keytab as kudu/127.6.152.62@KRBTEST.COM (short username kudu)
I20260928 23:54:11.702919 29428 webserver.cc:460] Webserver started at http://127.6.152.62:38295/ using document root <none> and password file <none>
I20260928 23:54:11.703476 29428 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:11.703564 29428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:11.703781 29428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:11.705840 29428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-0/data/instance:
uuid: "f19da64779f54715b078da81ed9b69ee"
format_stamp: "Formatted at 2026-09-28 23:54:11 on dist-test-slave-p5bw"
server_key: "e444293e0b3fb89a151dd9331aa44504"
server_key_iv: "3737d5efa906f23c14653b9faf221892"
server_key_version: "encryptionkey@0"
I20260928 23:54:11.706406 29428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-0/wal/instance:
uuid: "f19da64779f54715b078da81ed9b69ee"
format_stamp: "Formatted at 2026-09-28 23:54:11 on dist-test-slave-p5bw"
server_key: "e444293e0b3fb89a151dd9331aa44504"
server_key_iv: "3737d5efa906f23c14653b9faf221892"
server_key_version: "encryptionkey@0"
I20260928 23:54:11.709985 29428 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:54:11.712261 29443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:11.713454 29428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:54:11.713588 29428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-0/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-0/wal
uuid: "f19da64779f54715b078da81ed9b69ee"
format_stamp: "Formatted at 2026-09-28 23:54:11 on dist-test-slave-p5bw"
server_key: "e444293e0b3fb89a151dd9331aa44504"
server_key_iv: "3737d5efa906f23c14653b9faf221892"
server_key_version: "encryptionkey@0"
I20260928 23:54:11.713706 29428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:11.738036 29428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:11.741364 29428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:11.741567 29428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:11.748498 29428 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.62:45089
I20260928 23:54:11.748793 29495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.62:45089 every 8 connection(s)
I20260928 23:54:11.749538 29428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-0/data/info.pb
I20260928 23:54:11.750263 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 29428
I20260928 23:54:11.750396 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-0/wal/instance
I20260928 23:54:11.750705 6752 external_mini_cluster.cc:1428] Setting key ce6e0314211592b03f37f319308e6f2e
I20260928 23:54:11.752982 29496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:11.756560 29496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45089 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.152.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.61@KRBTEST.COM" created.
Sep 28 23:54:11 dist-test-slave-p5bw krb5kdc[29415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639651, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:54:11.771792 29500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:11.760250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54770 (local address 127.6.152.62:45089)
0928 23:54:11.760538 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:11.760548 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:11.760590 (+ 42us) server_negotiation.cc:407] Connection header received
0928 23:54:11.760973 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:11.760978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:11.761160 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:11.761354 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:11.762165 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:11.763089 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:11.764105 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:11.764529 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:11.764699 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:11.767007 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:11.767015 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:11.767030 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:11.767059 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:11.768728 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:11.769333 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:11.769337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:11.769339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:11.769441 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:11.769828 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:11.769830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:11.769831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:11.770176 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:11.770279 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:11.770781 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 23:54:11.771003 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":191,"thread_start_us":86,"threads_started":1}
I20260928 23:54:11.771795 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:11.759244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45089 (local address 127.0.0.1:54770)
0928 23:54:11.759744 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:11.759785 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:11.760660 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:11.761342 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:11.761346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:11.761767 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:11.762003 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:11.762016 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:11.763210 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:11.763213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:11.763984 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:11.763993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:11.764696 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:11.764700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:11.764795 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:11.765396 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:11.765421 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:11.766849 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:11.768869 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:11.768881 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:11.768894 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:11.769224 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:11.769559 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:11.769563 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:11.769565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:11.769719 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:11.770300 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:11.770310 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:11.770506 (+ 196us) client_negotiation.cc:769] Sending connection context
0928 23:54:11.770707 (+ 201us) client_negotiation.cc:240] Negotiation successful
0928 23:54:11.771107 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":81,"thread_start_us":141,"threads_started":1}
I20260928 23:54:11.774482 29496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.61" port: 37487 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:54:11.775251 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:11.774908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:37487 (local address 127.0.0.1:34790)
0928 23:54:11.775070 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:11.775163 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.61:37487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 23:54:11.775475 29444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.152.61:37487: connect: Connection refused (error 111)
I20260928 23:54:11.776280 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:11.776090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:37487 (local address 127.0.0.1:34792)
0928 23:54:11.776184 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:11.776225 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.61:37487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 23:54:11.776422 29496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.61:37487: Network error: Client connection negotiation failed: client connection to 127.6.152.61:37487: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.61.keytab.
Entry for principal kudu/127.6.152.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.61.keytab.
I20260928 23:54:11.781184 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.61:37487
--webserver_interface=127.6.152.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.61.keytab
--principal=kudu/127.6.152.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:46645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:45089,127.6.152.61:37487,127.6.152.60:41187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:54:11.825670 29496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.61" port: 37487 } attempt: 1
I20260928 23:54:11.826582 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:11.826216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:37487 (local address 127.0.0.1:34806)
0928 23:54:11.826419 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:11.826492 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.61:37487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 23:54:11.827299 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:11.827085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:37487 (local address 127.0.0.1:34810)
0928 23:54:11.827188 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:11.827237 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.61:37487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 23:54:11.827430 29496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.61:37487: Network error: Client connection negotiation failed: client connection to 127.6.152.61:37487: connect: Connection refused (error 111)
W20260928 23:54:11.895169 29503 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:54:11.895422 29503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:11.895478 29503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:11.895571 29503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 23:54:11.895594 29496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.61" port: 37487 } attempt: 2
I20260928 23:54:11.896446 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:11.896124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:37487 (local address 127.0.0.1:34814)
0928 23:54:11.896282 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:11.896360 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.61:37487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 23:54:11.897183 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:11.896973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:37487 (local address 127.0.0.1:34816)
0928 23:54:11.897076 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:11.897123 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.61:37487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 23:54:11.897333 29496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.61:37487: Network error: Client connection negotiation failed: client connection to 127.6.152.61:37487: connect: Connection refused (error 111)
W20260928 23:54:11.900035 29503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:54:11.900084 29503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:11.900123 29503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:11.900156 29503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:54:11.900189 29503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:54:11.905364 29503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:46645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:45089,127.6.152.61:37487,127.6.152.60:41187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.61.keytab
--principal=kudu/127.6.152.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.61:37487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.152.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:11.906304 29503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:11.907264 29503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:54:11.915602 29508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:11.915680 29503 server_base.cc:1034] running on GCE node
W20260928 23:54:11.915616 29509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:11.915614 29511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:11.916255 29503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:11.917376 29503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:11.918550 29503 hybrid_clock.cc:648] HybridClock initialized: now 1790639651918529 us; error 43 us; skew 500 ppm
Sep 28 23:54:11 dist-test-slave-p5bw krb5kdc[29415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639651, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:11.921098 29503 init.cc:377] Logged in from keytab as kudu/127.6.152.61@KRBTEST.COM (short username kudu)
I20260928 23:54:11.921905 29503 webserver.cc:460] Webserver started at http://127.6.152.61:37505/ using document root <none> and password file <none>
I20260928 23:54:11.922376 29503 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:11.922431 29503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:11.922578 29503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:11.924439 29503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-1/data/instance:
uuid: "a238a1ad7d9347a5ad602a6d027a0922"
format_stamp: "Formatted at 2026-09-28 23:54:11 on dist-test-slave-p5bw"
server_key: "7648336515476f8a9464f37ed45d86e1"
server_key_iv: "f085fb7c83191f5a35c93cd246d8a1e5"
server_key_version: "encryptionkey@0"
I20260928 23:54:11.924988 29503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-1/wal/instance:
uuid: "a238a1ad7d9347a5ad602a6d027a0922"
format_stamp: "Formatted at 2026-09-28 23:54:11 on dist-test-slave-p5bw"
server_key: "7648336515476f8a9464f37ed45d86e1"
server_key_iv: "f085fb7c83191f5a35c93cd246d8a1e5"
server_key_version: "encryptionkey@0"
I20260928 23:54:11.928344 29503 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:54:11.930663 29518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:11.931898 29503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:54:11.932018 29503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-1/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-1/wal
uuid: "a238a1ad7d9347a5ad602a6d027a0922"
format_stamp: "Formatted at 2026-09-28 23:54:11 on dist-test-slave-p5bw"
server_key: "7648336515476f8a9464f37ed45d86e1"
server_key_iv: "f085fb7c83191f5a35c93cd246d8a1e5"
server_key_version: "encryptionkey@0"
I20260928 23:54:11.932116 29503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:11.952051 29503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:11.955035 29503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:11.955240 29503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:11.962291 29503 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.61:37487
I20260928 23:54:11.962659 29570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.61:37487 every 8 connection(s)
I20260928 23:54:11.963311 29503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-1/data/info.pb
I20260928 23:54:11.966070 29571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:11.966974 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 29503
I20260928 23:54:11.967104 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-1/wal/instance
I20260928 23:54:11.967442 6752 external_mini_cluster.cc:1428] Setting key 5c62194f3f6d45a0be4ed954fe77accb
I20260928 23:54:11.970825 29571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45089 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.152.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.152.60@KRBTEST.COM" created.
Sep 28 23:54:11 dist-test-slave-p5bw krb5kdc[29415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639651, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.61@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:54:11.985862 29500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:11.972802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54780 (local address 127.6.152.62:45089)
0928 23:54:11.972981 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:11.972985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:11.974010 (+ 1025us) server_negotiation.cc:407] Connection header received
0928 23:54:11.974918 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:11.974922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:11.974985 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:11.975088 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:11.976367 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:11.977036 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:11.978563 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:11.979070 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:11.979216 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:11.981685 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:11.981692 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:11.981699 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:11.981747 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:11.983609 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:11.984108 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:11.984114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:11.984120 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:11.984220 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:11.984550 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:11.984554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:11.984559 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:11.984853 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:11.984971 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:11.985494 (+ 523us) server_negotiation.cc:299] Negotiation successful
0928 23:54:11.985635 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:54:11.986464 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:11.973323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45089 (local address 127.0.0.1:54780)
0928 23:54:11.973848 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:11.973889 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:11.974704 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:11.975257 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:11.975265 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:11.975669 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:11.976140 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:11.976152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:11.977193 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:11.977196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:11.978426 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:11.978436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:11.979224 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:11.979227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:11.979397 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:11.980029 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:11.980046 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:11.981508 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:11.983736 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:11.983741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:11.983752 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:11.983999 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:11.984322 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:11.984324 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:11.984326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:11.984451 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:11.984965 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:11.984970 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:11.985194 (+ 224us) client_negotiation.cc:769] Sending connection context
0928 23:54:11.985381 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 23:54:11.985618 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":135,"threads_started":1}
I20260928 23:54:11.987978 29571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.61" port: 37487 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:54:11.988476 29496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.61" port: 37487 } attempt: 3
Sep 28 23:54:11 dist-test-slave-p5bw krb5kdc[29415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639651, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.61@KRBTEST.COM for kudu/127.6.152.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.152.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.60.keytab.
Entry for principal kudu/127.6.152.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.60.keytab.
Sep 28 23:54:11 dist-test-slave-p5bw krb5kdc[29415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639651, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for kudu/127.6.152.61@KRBTEST.COM
I20260928 23:54:11.995488 6752 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
/tmp/dist-test-taskXX9CKr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.60:41187
--webserver_interface=127.6.152.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.60.keytab
--principal=kudu/127.6.152.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.152.20:46645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.152.62:45089,127.6.152.61:37487,127.6.152.60:41187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:54:11.999408 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:11.988915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:37487 (local address 127.0.0.1:34830)
0928 23:54:11.989033 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:11.989049 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:11.989123 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:11.989515 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:11.989518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:11.989708 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:11.989822 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:11.989828 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:11.990898 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:11.990903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:11.992012 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:11.992022 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:11.992636 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:11.992640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:11.992749 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:11.993166 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:11.993182 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:11.995117 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:11.997769 (+ 2652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:11.997772 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:11.997776 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:11.998043 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:11.998345 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:11.998349 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:11.998351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:11.998437 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:11.998892 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:11.998895 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:11.998990 (+ 95us) client_negotiation.cc:769] Sending connection context
0928 23:54:11.999106 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 23:54:11.999236 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 23:54:11.999504 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:11.988480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:37487 (local address 127.0.0.1:34828)
0928 23:54:11.988609 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:11.988624 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:11.988714 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:11.989441 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:11.989444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:11.989638 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:11.989767 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:11.989779 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:11.990927 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:11.990931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:11.991521 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:11.991530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:11.992238 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:11.992242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:11.992409 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:11.993087 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:11.993101 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:11.994626 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:11.997766 (+ 3140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:11.997769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:11.997770 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:11.997972 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:11.998318 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:11.998324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:11.998329 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:11.998414 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:11.998921 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:11.998927 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:11.999015 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 23:54:11.999098 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 23:54:11.999246 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":28}
I20260928 23:54:11.999519 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:11.988965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34830 (local address 127.6.152.61:37487)
0928 23:54:11.989258 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:11.989281 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:11.989298 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:54:11.989338 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:11.989344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:11.989396 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:11.989500 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:11.989934 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:11.990740 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:11.992131 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:11.992458 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:11.992632 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:11.995270 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:11.995275 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:11.995279 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:11.995304 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:11.997620 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:11.998153 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:11.998157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:11.998159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:11.998251 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:11.998527 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:11.998532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:11.998533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:11.998780 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:11.998921 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:11.999190 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 23:54:11.999326 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":207,"spinlock_wait_cycles":2816,"thread_start_us":99,"threads_started":1}
I20260928 23:54:12.000144 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:11.988615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34828 (local address 127.6.152.61:37487)
0928 23:54:11.989000 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:11.989013 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:11.989035 (+ 22us) server_negotiation.cc:407] Connection header received
0928 23:54:11.989083 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:11.989089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:11.989317 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:11.989465 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:11.989891 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:11.990823 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:11.991646 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:11.992118 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:11.992253 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:11.994795 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:11.994808 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:11.994825 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:11.994854 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:11.997651 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:11.998091 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:11.998100 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:11.998105 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:11.998200 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:11.998508 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:11.998514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:11.998518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:11.998818 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:11.999693 (+ 875us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:11.999774 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 23:54:11.999900 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":234,"server-negotiator.queue_time_us":286,"spinlock_wait_cycles":7296,"thread_start_us":145,"threads_started":1}
I20260928 23:54:12.001614 29571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:54:12.001771 29496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:54:12.002427 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.002173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:41187 (local address 127.0.0.1:59166)
0928 23:54:12.002284 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:12.002355 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 23:54:12.002481 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.002251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:41187 (local address 127.0.0.1:59172)
0928 23:54:12.002365 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:12.002412 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 23:54:12.002629 29522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.152.60:41187: connect: Connection refused (error 111)
I20260928 23:54:12.003160 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.002946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:41187 (local address 127.0.0.1:59174)
0928 23:54:12.003063 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:12.003106 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 23:54:12.003365 29496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.60:41187: Network error: Client connection negotiation failed: client connection to 127.6.152.60:41187: connect: Connection refused (error 111)
I20260928 23:54:12.003775 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.003613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:41187 (local address 127.0.0.1:59182)
0928 23:54:12.003693 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:12.003732 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 23:54:12.003966 29571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.60:41187: Network error: Client connection negotiation failed: client connection to 127.6.152.60:41187: connect: Connection refused (error 111)
I20260928 23:54:12.034547 29496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } attempt: 1
I20260928 23:54:12.035427 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.035094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:41187 (local address 127.0.0.1:59198)
0928 23:54:12.035236 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:12.035329 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 23:54:12.036193 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.035977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:41187 (local address 127.0.0.1:59210)
0928 23:54:12.036085 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:12.036131 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 23:54:12.036319 29496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.60:41187: Network error: Client connection negotiation failed: client connection to 127.6.152.60:41187: connect: Connection refused (error 111)
I20260928 23:54:12.053197 29571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } attempt: 1
I20260928 23:54:12.053992 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.053731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:41187 (local address 127.0.0.1:59224)
0928 23:54:12.053866 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:12.053921 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 23:54:12.054710 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.054503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:41187 (local address 127.0.0.1:59236)
0928 23:54:12.054609 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:12.054658 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 23:54:12.054821 29571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.60:41187: Network error: Client connection negotiation failed: client connection to 127.6.152.60:41187: connect: Connection refused (error 111)
W20260928 23:54:12.107631 29578 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:54:12.107966 29578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:54:12.108039 29578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:54:12.108119 29578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 23:54:12.111476 29496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } attempt: 2
I20260928 23:54:12.112313 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.112003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:41187 (local address 127.0.0.1:59250)
0928 23:54:12.112170 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:12.112233 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260928 23:54:12.112306 29578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:54:12.112381 29578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:54:12.112417 29578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:54:12.112463 29578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:54:12.112504 29578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:54:12.113035 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.112806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:41187 (local address 127.0.0.1:59266)
0928 23:54:12.112937 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:12.112986 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 23:54:12.113160 29496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.60:41187: Network error: Client connection negotiation failed: client connection to 127.6.152.60:41187: connect: Connection refused (error 111)
I20260928 23:54:12.117887 29578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.152.20:46645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.152.62:45089,127.6.152.61:37487,127.6.152.60:41187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/krb5kdc/kudu_127.6.152.60.keytab
--principal=kudu/127.6.152.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.152.60:41187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.152.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:54:12.118892 29578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:54:12.119874 29578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:54:12.122984 29571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } attempt: 2
I20260928 23:54:12.123775 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.123500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:41187 (local address 127.0.0.1:59270)
0928 23:54:12.123629 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:12.123683 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 23:54:12.124526 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.124240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:41187 (local address 127.0.0.1:59274)
0928 23:54:12.124404 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:12.124458 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.152.60:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260928 23:54:12.124675 29571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.152.60:41187: Network error: Client connection negotiation failed: client connection to 127.6.152.60:41187: connect: Connection refused (error 111)
W20260928 23:54:12.128623 29584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:12.128655 29587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:54:12.128598 29585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:54:12.128724 29578 server_base.cc:1034] running on GCE node
I20260928 23:54:12.129375 29578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:54:12.130434 29578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:54:12.131613 29578 hybrid_clock.cc:648] HybridClock initialized: now 1790639652131602 us; error 31 us; skew 500 ppm
Sep 28 23:54:12 dist-test-slave-p5bw krb5kdc[29415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639652, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:54:12.134356 29578 init.cc:377] Logged in from keytab as kudu/127.6.152.60@KRBTEST.COM (short username kudu)
I20260928 23:54:12.135294 29578 webserver.cc:460] Webserver started at http://127.6.152.60:40087/ using document root <none> and password file <none>
I20260928 23:54:12.135911 29578 fs_manager.cc:359] Metadata directory not provided
I20260928 23:54:12.136008 29578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:54:12.136253 29578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:54:12.138244 29578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-2/data/instance:
uuid: "6f43d9af09fa4e0da245143e30eb9fc9"
format_stamp: "Formatted at 2026-09-28 23:54:12 on dist-test-slave-p5bw"
server_key: "a7ed590a4b11dbea829c357e6bd11e2d"
server_key_iv: "861a213d7544acb5d381c7d5428f6cb0"
server_key_version: "encryptionkey@0"
I20260928 23:54:12.138737 29578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-2/wal/instance:
uuid: "6f43d9af09fa4e0da245143e30eb9fc9"
format_stamp: "Formatted at 2026-09-28 23:54:12 on dist-test-slave-p5bw"
server_key: "a7ed590a4b11dbea829c357e6bd11e2d"
server_key_iv: "861a213d7544acb5d381c7d5428f6cb0"
server_key_version: "encryptionkey@0"
I20260928 23:54:12.142220 29578 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:54:12.144541 29594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:54:12.145736 29578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:54:12.145870 29578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-2/data,/tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-2/wal
uuid: "6f43d9af09fa4e0da245143e30eb9fc9"
format_stamp: "Formatted at 2026-09-28 23:54:12 on dist-test-slave-p5bw"
server_key: "a7ed590a4b11dbea829c357e6bd11e2d"
server_key_iv: "861a213d7544acb5d381c7d5428f6cb0"
server_key_version: "encryptionkey@0"
I20260928 23:54:12.145973 29578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:54:12.166313 29578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:54:12.169615 29578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:54:12.169816 29578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:54:12.176278 29578 rpc_server.cc:307] RPC server started. Bound to: 127.6.152.60:41187
I20260928 23:54:12.176570 29646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.152.60:41187 every 8 connection(s)
I20260928 23:54:12.177248 29578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-2/data/info.pb
I20260928 23:54:12.179987 29647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:54:12.182204 6752 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu as pid 29578
I20260928 23:54:12.182345 6752 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXX9CKr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639374546653-6752-0/minicluster-data/master-2/wal/instance
I20260928 23:54:12.182672 6752 external_mini_cluster.cc:1428] Setting key 8dc77320613bf1c0a8b61f5441fb3407
I20260928 23:54:12.184414 29647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45089 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:54:12 dist-test-slave-p5bw krb5kdc[29415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.60@KRBTEST.COM
Sep 28 23:54:12 dist-test-slave-p5bw krb5kdc[29415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639652, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.60@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:54:12.196595 29500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.186905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54790 (local address 127.6.152.62:45089)
0928 23:54:12.187053 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:12.187058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:12.187312 (+ 254us) server_negotiation.cc:407] Connection header received
0928 23:54:12.187442 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:12.187446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:12.187495 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:12.187579 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:12.188205 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.188676 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.189731 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.190000 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.190118 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:12.192938 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:12.192942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:12.192944 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:12.192967 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:12.194571 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:12.195150 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:12.195153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:12.195155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:12.195297 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:12.195601 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:12.195604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:12.195605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:12.195845 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:12.195950 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:12.196347 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 23:54:12.196477 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 23:54:12.197467 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.183976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59288 (local address 127.6.152.60:41187)
0928 23:54:12.184385 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:12.184403 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:12.184445 (+ 42us) server_negotiation.cc:407] Connection header received
0928 23:54:12.185242 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:12.185267 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:12.185687 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:12.186039 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:12.186818 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.188077 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.189041 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.189553 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.189725 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:12.192327 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:12.192334 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:12.192347 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:12.192372 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:12.194301 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:12.194824 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:12.194831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:12.194834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:12.194950 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:12.195252 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:12.195254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:12.195255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:12.195723 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:12.195879 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:12.196159 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 23:54:12.196495 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":272,"thread_start_us":108,"threads_started":1}
I20260928 23:54:12.197467 29651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.186801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.62:45089 (local address 127.0.0.1:54790)
0928 23:54:12.187178 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:12.187227 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:12.187341 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:12.187629 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:12.187633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:12.187938 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:12.188061 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:12.188071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.188792 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.188796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:12.189575 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:12.189585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.190163 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.190167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:12.190268 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:12.190757 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:12.190785 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:12.192790 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:12.194698 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:12.194705 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:12.194712 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:12.195049 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:12.195389 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:12.195392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:12.195394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:12.195485 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:12.195956 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:12.195961 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:12.196108 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 23:54:12.196248 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 23:54:12.196543 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":161,"spinlock_wait_cycles":9472,"thread_start_us":148,"threads_started":1}
I20260928 23:54:12.198658 29647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.61" port: 37487 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:54:12 dist-test-slave-p5bw krb5kdc[29415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639652, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.60@KRBTEST.COM for kudu/127.6.152.61@KRBTEST.COM
Sep 28 23:54:12 dist-test-slave-p5bw krb5kdc[29415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.62@KRBTEST.COM
I20260928 23:54:12.207043 29651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.199130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.61:37487 (local address 127.0.0.1:34832)
0928 23:54:12.199276 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:12.199289 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:12.199355 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:12.199639 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:12.199642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:12.199815 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:12.199918 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:12.199925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.200661 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.200666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:12.201245 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:12.201253 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.201739 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.201743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:12.201824 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:12.202279 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:12.202292 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:12.203772 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:12.205654 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:12.205658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:12.205660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:12.205932 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:12.206198 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:12.206201 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:12.206203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:12.206292 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:12.206620 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:12.206625 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:12.206717 (+ 92us) client_negotiation.cc:769] Sending connection context
0928 23:54:12.206831 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 23:54:12.206938 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 23:54:12.207217 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.199245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34832 (local address 127.6.152.61:37487)
0928 23:54:12.199371 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:12.199374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:12.199424 (+ 50us) server_negotiation.cc:407] Connection header received
0928 23:54:12.199475 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:12.199477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:12.199530 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:12.199605 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:12.200042 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.200555 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.201375 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.201631 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.201743 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:12.203904 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:12.203907 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:12.203909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:12.203931 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:12.205538 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:12.206033 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:12.206035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:12.206037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:12.206108 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:12.206392 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:12.206394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:12.206395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:12.206546 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:12.206617 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:12.206909 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 23:54:12.207030 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:54:12.207882 29647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:54:12.208592 29500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.199872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54796 (local address 127.6.152.62:45089)
0928 23:54:12.199983 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:12.199987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:12.199999 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:54:12.200092 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:12.200095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:12.200131 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:12.200220 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:12.200791 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.201306 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.202063 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.202290 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.202394 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:12.205068 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:12.205072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:12.205074 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:12.205097 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:12.206696 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:12.207276 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:12.207280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:12.207281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:12.207368 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:12.207688 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:12.207691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:12.207693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:12.207879 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:12.208085 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:12.208252 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 23:54:12.208468 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 23:54:12 dist-test-slave-p5bw krb5kdc[29415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639652, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.60@KRBTEST.COM for kudu/127.6.152.60@KRBTEST.COM
I20260928 23:54:12.212318 29496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } attempt: 3
Sep 28 23:54:12 dist-test-slave-p5bw krb5kdc[29415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.152.61@KRBTEST.COM
I20260928 23:54:12.215860 29571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } attempt: 3
I20260928 23:54:12.216403 29651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.208347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:41187 (local address 127.0.0.1:59298)
0928 23:54:12.208445 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:12.208459 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:12.208523 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:12.208981 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:12.208983 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:12.209145 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:12.209221 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:12.209227 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.210070 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.210073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:12.210595 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:12.210604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.211012 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.211016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:12.211093 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:12.211543 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:12.211554 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:12.212750 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:12.214667 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:12.214673 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:12.214677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:12.214898 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:12.215447 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:12.215450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:12.215452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:12.215548 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:12.215954 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:12.215957 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:12.216036 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 23:54:12.216140 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 23:54:12.216253 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 23:54:12.216913 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.208387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59298 (local address 127.6.152.60:41187)
0928 23:54:12.208779 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:12.208783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:12.208794 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:54:12.208844 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:12.208846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:12.208886 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:12.208973 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:12.209330 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.209991 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.210702 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.210916 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.210988 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:12.212905 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:12.212908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:12.212910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:12.212931 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:12.214556 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:12.215000 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:12.215003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:12.215007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:12.215343 (+ 336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:12.215624 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:12.215630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:12.215632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:12.215794 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:12.215919 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:12.216606 (+ 687us) server_negotiation.cc:299] Negotiation successful
0928 23:54:12.216711 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329}
Sep 28 23:54:12 dist-test-slave-p5bw krb5kdc[29415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639651, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.62@KRBTEST.COM for kudu/127.6.152.60@KRBTEST.COM
I20260928 23:54:12.219347 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.209682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34844 (local address 127.6.152.61:37487)
0928 23:54:12.209772 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:12.209776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:12.209789 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:54:12.209871 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:12.209873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:12.209903 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:12.209962 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:12.210548 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.211150 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.212125 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.212343 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.212447 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:12.214791 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:12.214795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:12.214801 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:12.214828 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:12.216559 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:12.217155 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:12.217158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:12.217160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:12.217238 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:12.217608 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:12.217612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:12.217613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:12.218163 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:12.218345 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:12.219035 (+ 690us) server_negotiation.cc:299] Negotiation successful
0928 23:54:12.219194 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":27}
I20260928 23:54:12.220460 29647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f43d9af09fa4e0da245143e30eb9fc9: Bootstrap starting.
Sep 28 23:54:12 dist-test-slave-p5bw krb5kdc[29415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639651, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.152.61@KRBTEST.COM for kudu/127.6.152.60@KRBTEST.COM
I20260928 23:54:12.223393 6752 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:54:12.223399 29647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f43d9af09fa4e0da245143e30eb9fc9: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:12.224236 29647 log.cc:824] T 00000000000000000000000000000000 P 6f43d9af09fa4e0da245143e30eb9fc9: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:12.224742 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.212643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:41187 (local address 127.0.0.1:59304)
0928 23:54:12.212831 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:12.212846 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:12.212920 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:12.214244 (+ 1324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:12.214249 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:12.214427 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:12.214499 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:12.214504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.215935 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.215939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:12.216746 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:12.216758 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.217455 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.217462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:12.217554 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:12.217978 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:12.217993 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:12.219442 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:12.223293 (+ 3851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:12.223296 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:12.223298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:12.223586 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:12.223844 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:12.223848 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:12.223849 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:12.223938 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:12.224281 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:12.224284 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:12.224383 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 23:54:12.224477 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 23:54:12.224608 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 23:54:12.225252 29652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.212721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59304 (local address 127.6.152.60:41187)
0928 23:54:12.214007 (+ 1286us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:12.214016 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:12.214029 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:54:12.214062 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:12.214065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:12.214111 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:12.214200 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:12.214610 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.215800 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.216911 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.217320 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.217847 (+ 527us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:12.219876 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:12.219880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:12.219882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:12.219905 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:12.223155 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:12.223686 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:12.223688 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:12.223689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:12.223753 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:12.224027 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:12.224032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:12.224034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:12.224185 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:12.224288 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:12.224968 (+ 680us) server_negotiation.cc:299] Negotiation successful
0928 23:54:12.225077 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":1209,"thread_start_us":152,"threads_started":1}
I20260928 23:54:12.224589 29470 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f19da64779f54715b078da81ed9b69ee"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54796
I20260928 23:54:12.226758 29647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f43d9af09fa4e0da245143e30eb9fc9: No bootstrap required, opened a new log
I20260928 23:54:12.227700 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.216206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.152.60:41187 (local address 127.0.0.1:59320)
0928 23:54:12.216446 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 23:54:12.216462 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:54:12.216538 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:54:12.217381 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:54:12.217384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:54:12.217541 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:54:12.217629 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:12.217637 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.218533 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.218536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:12.219315 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:54:12.219327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.219869 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.219872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:54:12.219948 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:12.220419 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:54:12.220432 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:54:12.221750 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:54:12.226094 (+ 4344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:12.226100 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:12.226102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:12.226375 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:12.226699 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:54:12.226702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:54:12.226704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:54:12.226804 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:54:12.227239 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:54:12.227242 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:54:12.227338 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 23:54:12.227434 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 23:54:12.227556 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":3}
I20260928 23:54:12.228209 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.216655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59320 (local address 127.6.152.60:41187)
0928 23:54:12.217044 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:12.217049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:12.217064 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:54:12.217096 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:12.217099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:12.217132 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:12.217205 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:12.217719 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.218409 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.219460 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.219733 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.220130 (+ 397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:12.221880 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:12.221885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:12.221890 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:12.221913 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:12.224984 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:12.226513 (+ 1529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:12.226516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:12.226518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:12.226597 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:12.226915 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:12.226918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:12.226920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:12.227111 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:12.227196 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:12.227964 (+ 768us) server_negotiation.cc:299] Negotiation successful
0928 23:54:12.228063 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":325,"thread_start_us":79,"threads_started":1}
I20260928 23:54:12.228452 29496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee: Bootstrap starting.
I20260928 23:54:12.230860 29571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a238a1ad7d9347a5ad602a6d027a0922: Bootstrap starting.
I20260928 23:54:12.230904 29496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:12.230791 29647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f43d9af09fa4e0da245143e30eb9fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19da64779f54715b078da81ed9b69ee" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45089 } } peers { permanent_uuid: "a238a1ad7d9347a5ad602a6d027a0922" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 37487 } } peers { permanent_uuid: "6f43d9af09fa4e0da245143e30eb9fc9" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } }
I20260928 23:54:12.231002 29647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f43d9af09fa4e0da245143e30eb9fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f43d9af09fa4e0da245143e30eb9fc9, State: Initialized, Role: FOLLOWER
I20260928 23:54:12.231300 29647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f43d9af09fa4e0da245143e30eb9fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19da64779f54715b078da81ed9b69ee" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45089 } } peers { permanent_uuid: "a238a1ad7d9347a5ad602a6d027a0922" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 37487 } } peers { permanent_uuid: "6f43d9af09fa4e0da245143e30eb9fc9" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } }
I20260928 23:54:12.231679 29496 log.cc:824] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:12.231812 29655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f43d9af09fa4e0da245143e30eb9fc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19da64779f54715b078da81ed9b69ee" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45089 } } peers { permanent_uuid: "a238a1ad7d9347a5ad602a6d027a0922" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 37487 } } peers { permanent_uuid: "6f43d9af09fa4e0da245143e30eb9fc9" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } } }
I20260928 23:54:12.232023 29655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f43d9af09fa4e0da245143e30eb9fc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:54:12.232520 29647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f43d9af09fa4e0da245143e30eb9fc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:54:12.233160 29571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a238a1ad7d9347a5ad602a6d027a0922: Neither blocks nor log segments found. Creating new log.
I20260928 23:54:12.233729 29496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee: No bootstrap required, opened a new log
I20260928 23:54:12.233971 29571 log.cc:824] T 00000000000000000000000000000000 P a238a1ad7d9347a5ad602a6d027a0922: Log is configured to *not* fsync() on all Append() calls
I20260928 23:54:12.236135 29571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a238a1ad7d9347a5ad602a6d027a0922: No bootstrap required, opened a new log
I20260928 23:54:12.236769 29496 raft_consensus.cc:348] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19da64779f54715b078da81ed9b69ee" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45089 } } peers { permanent_uuid: "a238a1ad7d9347a5ad602a6d027a0922" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 37487 } } peers { permanent_uuid: "6f43d9af09fa4e0da245143e30eb9fc9" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } }
I20260928 23:54:12.237066 29496 raft_consensus.cc:729] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19da64779f54715b078da81ed9b69ee, State: Initialized, Role: FOLLOWER
I20260928 23:54:12.237406 29496 consensus_queue.cc:260] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19da64779f54715b078da81ed9b69ee" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45089 } } peers { permanent_uuid: "a238a1ad7d9347a5ad602a6d027a0922" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 37487 } } peers { permanent_uuid: "6f43d9af09fa4e0da245143e30eb9fc9" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } }
I20260928 23:54:12.238631 29496 sys_catalog.cc:564] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:54:12.238677 29470 raft_consensus.cc:482] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 23:54:12.238777 29470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:12.239979 29470 raft_consensus.cc:504] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19da64779f54715b078da81ed9b69ee" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45089 } } peers { permanent_uuid: "a238a1ad7d9347a5ad602a6d027a0922" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 37487 } } peers { permanent_uuid: "6f43d9af09fa4e0da245143e30eb9fc9" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } }
I20260928 23:54:12.237820 29578 master_runner.cc:186] Error getting master registration for 127.6.152.62:45089: OK, instance_id { permanent_uuid: "f19da64779f54715b078da81ed9b69ee" instance_seqno: 1790639651746575 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:54:12.240932 29470 leader_election.cc:288] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee [CANDIDATE]: Term 1 election: Requested vote from peers a238a1ad7d9347a5ad602a6d027a0922 (127.6.152.61:37487), 6f43d9af09fa4e0da245143e30eb9fc9 (127.6.152.60:41187)
I20260928 23:54:12.240990 29578 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:54:12.242412 29621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f19da64779f54715b078da81ed9b69ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6f43d9af09fa4e0da245143e30eb9fc9"
I20260928 23:54:12.242630 29621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f43d9af09fa4e0da245143e30eb9fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:12.244048 29571 raft_consensus.cc:348] T 00000000000000000000000000000000 P a238a1ad7d9347a5ad602a6d027a0922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19da64779f54715b078da81ed9b69ee" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45089 } } peers { permanent_uuid: "a238a1ad7d9347a5ad602a6d027a0922" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 37487 } } peers { permanent_uuid: "6f43d9af09fa4e0da245143e30eb9fc9" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } }
W20260928 23:54:12.244220 29681 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 23:54:12.244285 29571 raft_consensus.cc:729] T 00000000000000000000000000000000 P a238a1ad7d9347a5ad602a6d027a0922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a238a1ad7d9347a5ad602a6d027a0922, State: Initialized, Role: FOLLOWER
W20260928 23:54:12.244315 29681 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 23:54:12.244514 29449 tablet.cc:2369] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:54:12.244580 29571 consensus_queue.cc:260] T 00000000000000000000000000000000 P a238a1ad7d9347a5ad602a6d027a0922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19da64779f54715b078da81ed9b69ee" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45089 } } peers { permanent_uuid: "a238a1ad7d9347a5ad602a6d027a0922" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 37487 } } peers { permanent_uuid: "6f43d9af09fa4e0da245143e30eb9fc9" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } }
I20260928 23:54:12.245738 29571 sys_catalog.cc:564] T 00000000000000000000000000000000 P a238a1ad7d9347a5ad602a6d027a0922 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:54:12.245829 29544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f19da64779f54715b078da81ed9b69ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a238a1ad7d9347a5ad602a6d027a0922"
I20260928 23:54:12.246024 29544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a238a1ad7d9347a5ad602a6d027a0922 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:54:12.247279 29544 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a238a1ad7d9347a5ad602a6d027a0922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19da64779f54715b078da81ed9b69ee in term 1.
I20260928 23:54:12.247706 29444 leader_election.cc:302] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a238a1ad7d9347a5ad602a6d027a0922, f19da64779f54715b078da81ed9b69ee; no voters:
I20260928 23:54:12.249246 29621 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6f43d9af09fa4e0da245143e30eb9fc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19da64779f54715b078da81ed9b69ee in term 1.
W20260928 23:54:12.249876 29668 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6f43d9af09fa4e0da245143e30eb9fc9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:54:12.249944 29668 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:54:12.246795 29428 master_runner.cc:186] Error getting master registration for 127.6.152.61:37487: OK, instance_id { permanent_uuid: "a238a1ad7d9347a5ad602a6d027a0922" instance_seqno: 1790639651960317 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:54:12.250229 29428 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:54:12.250324 29670 sys_catalog.cc:455] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19da64779f54715b078da81ed9b69ee" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45089 } } peers { permanent_uuid: "a238a1ad7d9347a5ad602a6d027a0922" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 37487 } } peers { permanent_uuid: "6f43d9af09fa4e0da245143e30eb9fc9" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } } }
I20260928 23:54:12.250514 29670 sys_catalog.cc:458] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:54:12.250648 29670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:54:12.250766 29670 raft_consensus.cc:686] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee [term 1 LEADER]: Becoming Leader. State: Replica: f19da64779f54715b078da81ed9b69ee, State: Running, Role: LEADER
I20260928 23:54:12.251124 29670 consensus_queue.cc:237] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19da64779f54715b078da81ed9b69ee" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45089 } } peers { permanent_uuid: "a238a1ad7d9347a5ad602a6d027a0922" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 37487 } } peers { permanent_uuid: "6f43d9af09fa4e0da245143e30eb9fc9" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } }
I20260928 23:54:12.254046 29670 sys_catalog.cc:455] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader f19da64779f54715b078da81ed9b69ee. Latest consensus state: current_term: 1 leader_uuid: "f19da64779f54715b078da81ed9b69ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19da64779f54715b078da81ed9b69ee" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45089 } } peers { permanent_uuid: "a238a1ad7d9347a5ad602a6d027a0922" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 37487 } } peers { permanent_uuid: "6f43d9af09fa4e0da245143e30eb9fc9" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } } }
I20260928 23:54:12.254217 29670 sys_catalog.cc:458] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:12.262996 29687 sys_catalog.cc:455] T 00000000000000000000000000000000 P a238a1ad7d9347a5ad602a6d027a0922 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19da64779f54715b078da81ed9b69ee" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45089 } } peers { permanent_uuid: "a238a1ad7d9347a5ad602a6d027a0922" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 37487 } } peers { permanent_uuid: "6f43d9af09fa4e0da245143e30eb9fc9" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } } }
I20260928 23:54:12.263115 29687 sys_catalog.cc:458] T 00000000000000000000000000000000 P a238a1ad7d9347a5ad602a6d027a0922 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:54:12.263934 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.244799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34850 (local address 127.6.152.61:37487)
0928 23:54:12.244935 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:12.244940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:12.254650 (+ 9710us) server_negotiation.cc:407] Connection header received
0928 23:54:12.254844 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:12.254847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:12.254897 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:12.254949 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:12.255957 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.256748 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.258002 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.258350 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.258442 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:12.260276 (+ 1834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:12.260279 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:12.260282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:12.260305 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:12.261997 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:12.262478 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:12.262484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:12.262487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:12.262571 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:12.262904 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:12.262907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:12.262909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:12.263118 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:12.263246 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:12.263687 (+ 441us) server_negotiation.cc:299] Negotiation successful
0928 23:54:12.263794 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 23:54:12.265444 29704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:54:12.265763 29500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.245564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54808 (local address 127.6.152.62:45089)
0928 23:54:12.245660 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:12.245664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:12.257041 (+ 11377us) server_negotiation.cc:407] Connection header received
0928 23:54:12.257197 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:12.257201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:12.257245 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:12.257293 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:12.258180 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.258967 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.260092 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.260431 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.260498 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:12.262321 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:12.262324 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:12.262327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:12.262351 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:12.263967 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:12.264442 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:12.264445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:12.264447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:12.264526 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:12.264841 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:12.264844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:12.264846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:12.265037 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:12.265154 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:12.265503 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 23:54:12.265615 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 23:54:12.266311 29700 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a238a1ad7d9347a5ad602a6d027a0922: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:54:12.266387 29700 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:54:12.267030 29704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:54:12.271078 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:12.250944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.6.152.60:41187)
0928 23:54:12.251046 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:12.251050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:12.258432 (+ 7382us) server_negotiation.cc:407] Connection header received
0928 23:54:12.258594 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:12.258597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:12.258639 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:12.258687 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:12.259669 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.260425 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.261613 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:12.261986 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:12.262068 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:12.267660 (+ 5592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:54:12.267664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:54:12.267666 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:54:12.267691 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:54:12.269405 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:12.269865 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:12.269868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:12.269870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:12.269948 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:54:12.270220 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:54:12.270223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:54:12.270224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:54:12.270425 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:54:12.270526 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:12.270830 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 23:54:12.270934 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 23:54:12.282101 29621 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6f43d9af09fa4e0da245143e30eb9fc9 [term 1 FOLLOWER]: Refusing update from remote peer f19da64779f54715b078da81ed9b69ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:54:12.282908 29544 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a238a1ad7d9347a5ad602a6d027a0922 [term 1 FOLLOWER]: Refusing update from remote peer f19da64779f54715b078da81ed9b69ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:54:12.283299 29670 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f43d9af09fa4e0da245143e30eb9fc9" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:54:12.284257 29698 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238a1ad7d9347a5ad602a6d027a0922" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 37487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:54:12.288319 29687 sys_catalog.cc:455] T 00000000000000000000000000000000 P a238a1ad7d9347a5ad602a6d027a0922 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f19da64779f54715b078da81ed9b69ee. Latest consensus state: current_term: 1 leader_uuid: "f19da64779f54715b078da81ed9b69ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19da64779f54715b078da81ed9b69ee" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45089 } } peers { permanent_uuid: "a238a1ad7d9347a5ad602a6d027a0922" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 37487 } } peers { permanent_uuid: "6f43d9af09fa4e0da245143e30eb9fc9" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } } }
I20260928 23:54:12.288453 29687 sys_catalog.cc:458] T 00000000000000000000000000000000 P a238a1ad7d9347a5ad602a6d027a0922 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:54:12.291594 29655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f43d9af09fa4e0da245143e30eb9fc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f19da64779f54715b078da81ed9b69ee. Latest consensus state: current_term: 1 leader_uuid: "f19da64779f54715b078da81ed9b69ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19da64779f54715b078da81ed9b69ee" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45089 } } peers { permanent_uuid: "a238a1ad7d9347a5ad602a6d027a0922" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 37487 } } peers { permanent_uuid: "6f43d9af09fa4e0da245143e30eb9fc9" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } } }
I20260928 23:54:12.291708 29655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f43d9af09fa4e0da245143e30eb9fc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:54:12.302193 29670 sys_catalog.cc:455] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f19da64779f54715b078da81ed9b69ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19da64779f54715b078da81ed9b69ee" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45089 } } peers { permanent_uuid: "a238a1ad7d9347a5ad602a6d027a0922" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 37487 } } peers { permanent_uuid: "6f43d9af09fa4e0da245143e30eb9fc9" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } } }
I20260928 23:54:12.302315 29670 sys_catalog.cc:458] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:12.302423 29706 mvcc.cc:205] Tried to move back new op lower bound from 7334460015730987008 to 7334460015623766016. Current Snapshot: MvccSnapshot[applied={T|T < 7334460015730987008}]
I20260928 23:54:12.303663 29655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f43d9af09fa4e0da245143e30eb9fc9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f19da64779f54715b078da81ed9b69ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19da64779f54715b078da81ed9b69ee" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45089 } } peers { permanent_uuid: "a238a1ad7d9347a5ad602a6d027a0922" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 37487 } } peers { permanent_uuid: "6f43d9af09fa4e0da245143e30eb9fc9" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } } }
I20260928 23:54:12.303766 29655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f43d9af09fa4e0da245143e30eb9fc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:54:12.304986 29698 sys_catalog.cc:455] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f19da64779f54715b078da81ed9b69ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19da64779f54715b078da81ed9b69ee" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45089 } } peers { permanent_uuid: "a238a1ad7d9347a5ad602a6d027a0922" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 37487 } } peers { permanent_uuid: "6f43d9af09fa4e0da245143e30eb9fc9" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } } }
I20260928 23:54:12.305121 29698 sys_catalog.cc:458] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee [sys.catalog]: This master's current role is: LEADER
I20260928 23:54:12.306018 29687 sys_catalog.cc:455] T 00000000000000000000000000000000 P a238a1ad7d9347a5ad602a6d027a0922 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f19da64779f54715b078da81ed9b69ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19da64779f54715b078da81ed9b69ee" member_type: VOTER last_known_addr { host: "127.6.152.62" port: 45089 } } peers { permanent_uuid: "a238a1ad7d9347a5ad602a6d027a0922" member_type: VOTER last_known_addr { host: "127.6.152.61" port: 37487 } } peers { permanent_uuid: "6f43d9af09fa4e0da245143e30eb9fc9" member_type: VOTER last_known_addr { host: "127.6.152.60" port: 41187 } } }
I20260928 23:54:12.306133 29687 sys_catalog.cc:458] T 00000000000000000000000000000000 P a238a1ad7d9347a5ad602a6d027a0922 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:54:12.309780 29707 mvcc.cc:205] Tried to move back new op lower bound from 7334460015730987008 to 7334460015623766016. Current Snapshot: MvccSnapshot[applied={T|T < 7334460015730987008}]
I20260928 23:54:12.309962 29709 mvcc.cc:205] Tried to move back new op lower bound from 7334460015730987008 to 7334460015623766016. Current Snapshot: MvccSnapshot[applied={T|T < 7334460015730987008}]
I20260928 23:54:12.313282 29704 catalog_manager.cc:1347] Generated new cluster ID: 7637a65f3e4347b1a397bec0e1489bea
I20260928 23:54:12.313344 29704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:54:12.339773 29704 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:54:12.341526 29704 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:54:12.361720 29704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f19da64779f54715b078da81ed9b69ee: Generated new TSK 0
I20260928 23:54:12.362756 29704 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:54:12.396330 29500 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:12.393926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54818 (local address 127.6.152.62:45089)
0928 23:54:12.394058 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:12.394064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:12.395243 (+ 1179us) server_negotiation.cc:407] Connection header received
0928 23:54:12.395459 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:12.395462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:12.395517 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:12.395571 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:12.396200 (+ 629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54818: BlockingRecv error: recv got EOF from 127.0.0.1:54818 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 23:54:12.397910 29650 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:12.394667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59324 (local address 127.6.152.60:41187)
0928 23:54:12.394810 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:12.394814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:12.397025 (+ 2211us) server_negotiation.cc:407] Connection header received
0928 23:54:12.397192 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:12.397195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:12.397245 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:12.397296 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:12.397798 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59324: BlockingRecv error: recv got EOF from 127.0.0.1:59324 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 23:54:12.400009 29576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:12.398669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34854 (local address 127.6.152.61:37487)
0928 23:54:12.398782 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:12.398787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:12.399116 (+ 329us) server_negotiation.cc:407] Connection header received
0928 23:54:12.399287 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:12.399290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:12.399354 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:12.399409 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:12.399897 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34854: BlockingRecv error: recv got EOF from 127.0.0.1:34854 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 23:54:12.404577 29576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:12.403000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34860 (local address 127.6.152.61:37487)
0928 23:54:12.403093 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:12.403097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:12.403608 (+ 511us) server_negotiation.cc:407] Connection header received
0928 23:54:12.403828 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:12.403831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:12.403905 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:12.403959 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:12.404483 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34860: BlockingRecv error: recv got EOF from 127.0.0.1:34860 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 23:54:12.409865 29576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:12.408380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34868 (local address 127.6.152.61:37487)
0928 23:54:12.408481 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:12.408485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:12.408971 (+ 486us) server_negotiation.cc:407] Connection header received
0928 23:54:12.409150 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:12.409154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:12.409205 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:12.409258 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:12.409759 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34868: BlockingRecv error: recv got EOF from 127.0.0.1:34868 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 23:54:12.415784 29576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:12.414260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34870 (local address 127.6.152.61:37487)
0928 23:54:12.414371 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:12.414375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:12.414878 (+ 503us) server_negotiation.cc:407] Connection header received
0928 23:54:12.415072 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:12.415076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:12.415124 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:12.415176 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:12.415684 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34870: BlockingRecv error: recv got EOF from 127.0.0.1:34870 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 23:54:12.424057 29576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:12.422372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34886 (local address 127.6.152.61:37487)
0928 23:54:12.422521 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:12.422525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:12.423040 (+ 515us) server_negotiation.cc:407] Connection header received
0928 23:54:12.423275 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:12.423279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:12.423344 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:12.423400 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:12.423949 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34886: BlockingRecv error: recv got EOF from 127.0.0.1:34886 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260928 23:54:12.436982 29576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:12.434929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34890 (local address 127.6.152.61:37487)
0928 23:54:12.435074 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:12.435079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:12.435745 (+ 666us) server_negotiation.cc:407] Connection header received
0928 23:54:12.435971 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:12.435974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:12.436051 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:12.436112 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:12.436865 (+ 753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34890: BlockingRecv error: recv got EOF from 127.0.0.1:34890 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 23:54:12.456902 29576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:12.455636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34904 (local address 127.6.152.61:37487)
0928 23:54:12.455760 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:12.455765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:12.455996 (+ 231us) server_negotiation.cc:407] Connection header received
0928 23:54:12.456133 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:12.456136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:12.456190 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:12.456288 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:12.456756 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34904: BlockingRecv error: recv got EOF from 127.0.0.1:34904 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 23:54:12.492981 29576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:12.491671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34908 (local address 127.6.152.61:37487)
0928 23:54:12.491825 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:12.491830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:12.491997 (+ 167us) server_negotiation.cc:407] Connection header received
0928 23:54:12.492184 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:12.492186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:12.492236 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:12.492363 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:12.492838 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34908: BlockingRecv error: recv got EOF from 127.0.0.1:34908 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 23:54:12.560434 29576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:12.559330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.6.152.61:37487)
0928 23:54:12.559601 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:12.559605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:12.559622 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:54:12.559675 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:12.559678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:12.559722 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:12.559799 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:12.560257 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34922: BlockingRecv error: recv got EOF from 127.0.0.1:34922 (error 108)
Metrics: {"server-negotiator.queue_time_us":178}
W20260928 23:54:12.691803 29576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:12.690544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34934 (local address 127.6.152.61:37487)
0928 23:54:12.690717 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:12.690722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:12.690883 (+ 161us) server_negotiation.cc:407] Connection header received
0928 23:54:12.691062 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:12.691065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:12.691123 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:12.691215 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:12.691672 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34934: BlockingRecv error: recv got EOF from 127.0.0.1:34934 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260928 23:54:12.951181 29576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:54:12.950024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34936 (local address 127.6.152.61:37487)
0928 23:54:12.950147 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:12.950152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:12.950264 (+ 112us) server_negotiation.cc:407] Connection header received
0928 23:54:12.950441 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:12.950445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:12.950506 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:12.950597 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:54:12.951062 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34936: BlockingRecv error: recv got EOF from 127.0.0.1:34936 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:54:13.251252 29668 catalog_manager.cc:1259] Loaded cluster ID: 7637a65f3e4347b1a397bec0e1489bea
I20260928 23:54:13.251338 29668 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6f43d9af09fa4e0da245143e30eb9fc9: loading cluster ID for follower catalog manager: success
I20260928 23:54:13.252996 29668 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6f43d9af09fa4e0da245143e30eb9fc9: acquiring CA information for follower catalog manager: success
I20260928 23:54:13.254094 29668 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6f43d9af09fa4e0da245143e30eb9fc9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:54:13.267923 29700 catalog_manager.cc:1259] Loaded cluster ID: 7637a65f3e4347b1a397bec0e1489bea
I20260928 23:54:13.268014 29700 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a238a1ad7d9347a5ad602a6d027a0922: loading cluster ID for follower catalog manager: success
I20260928 23:54:13.269565 29700 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a238a1ad7d9347a5ad602a6d027a0922: acquiring CA information for follower catalog manager: success
I20260928 23:54:13.270588 29700 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a238a1ad7d9347a5ad602a6d027a0922: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:54:13.469002 29777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:13.465347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34938 (local address 127.6.152.61:37487)
0928 23:54:13.465565 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:13.465569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:13.465604 (+ 35us) server_negotiation.cc:407] Connection header received
0928 23:54:13.465795 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:13.465799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:13.465925 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:13.466037 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:54:13.466260 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:13.466799 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:13.467630 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:13.467862 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:13.467939 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:13.468195 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:54:13.468623 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:54:13.468712 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:13.468768 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 23:54:13.468874 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":77,"threads_started":1}
I20260928 23:54:13.475132 29783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:13.471813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59332 (local address 127.6.152.60:41187)
0928 23:54:13.472022 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:13.472026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:13.472039 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:54:13.472164 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:13.472167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:13.472278 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:13.472373 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:54:13.472522 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:13.473084 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:13.473858 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:13.474087 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:13.474165 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:13.474407 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:54:13.474828 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:54:13.474914 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:13.474968 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 23:54:13.475014 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":73,"threads_started":1}
I20260928 23:54:13.481433 29777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:13.477825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34946 (local address 127.6.152.61:37487)
0928 23:54:13.477941 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:13.477945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:13.478211 (+ 266us) server_negotiation.cc:407] Connection header received
0928 23:54:13.478456 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:13.478461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:13.478531 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:13.478602 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:54:13.478871 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:13.479416 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:13.480299 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:13.480631 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:13.480731 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:13.481004 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:54:13.481078 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:54:13.481157 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:13.481248 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 23:54:13.481297 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:54:13.482384 29783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:13.478242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59342 (local address 127.6.152.60:41187)
0928 23:54:13.478358 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:13.478362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:13.478540 (+ 178us) server_negotiation.cc:407] Connection header received
0928 23:54:13.478627 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:13.478632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:13.478680 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:13.478751 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:54:13.479240 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:13.480001 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:13.481221 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:13.481521 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:13.481618 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:13.481941 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:54:13.482014 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:54:13.482093 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:13.482219 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 23:54:13.482264 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:54:13.482635 29790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:54:13.477921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54828 (local address 127.6.152.62:45089)
0928 23:54:13.478605 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0928 23:54:13.478609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:54:13.478621 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:54:13.478706 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:54:13.478712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:54:13.478863 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:54:13.478994 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:54:13.479217 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:13.480019 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:13.481059 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:54:13.481406 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:54:13.481787 (+ 381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:54:13.481910 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:54:13.482224 (+ 314us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:54:13.482316 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 23:54:13.482391 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 23:54:13.482444 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":89,"threads_started":1}
I20260928 23:54:13.485637 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 29428
I20260928 23:54:13.491879 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 29503
I20260928 23:54:13.497434 6752 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXX9CKr/build/debug/bin/kudu with pid 29578
2026-09-28T23:54:13Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3088 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6191 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (278943 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST